Sep 11 14:15:11.816994 ip-10-0-131-57 systemd[1]: Starting Kubernetes Kubelet... Sep 11 14:15:12.133747 ip-10-0-131-57 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:15:12.133747 ip-10-0-131-57 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 14:15:12.133747 ip-10-0-131-57 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:15:12.133747 ip-10-0-131-57 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 14:15:12.133747 ip-10-0-131-57 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:15:12.135131 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.135045 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 14:15:12.138399 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138385 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:15:12.138399 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138400 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138404 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138408 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138411 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138414 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138417 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138420 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138423 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138426 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138429 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138431 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138442 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138445 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138447 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138450 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138453 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138456 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138458 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138461 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:15:12.138462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138464 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138466 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138469 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138472 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138475 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138478 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138480 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138483 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138486 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138488 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138491 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138494 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138497 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138499 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138502 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138504 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138507 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138510 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138513 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138515 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:15:12.138923 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138518 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138520 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138523 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138525 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138528 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138530 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138533 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138535 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138538 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138541 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138543 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138546 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138550 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138554 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138558 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138562 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138565 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138567 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138570 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:15:12.139397 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138573 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138576 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138578 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138581 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138583 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138586 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138589 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138591 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138594 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138596 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138599 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138602 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138604 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138607 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138609 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138612 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138614 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138617 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138620 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138622 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:15:12.139891 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138625 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138627 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138630 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138632 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138635 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138637 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138640 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138643 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138646 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138649 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138652 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138654 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138657 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138659 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138662 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138664 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138669 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138672 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138676 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.138680 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:15:12.140375 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139360 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139367 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139370 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139373 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139376 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139378 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139381 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139384 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139386 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139389 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139391 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139394 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139397 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139399 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139402 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139405 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139408 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139411 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139414 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:15:12.140873 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139417 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139420 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139423 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139425 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139430 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139433 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139436 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139439 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139442 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139444 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139447 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139450 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139453 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139455 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139458 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139469 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139472 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139475 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139478 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:15:12.141358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139481 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139484 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139487 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139490 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139492 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139495 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139497 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139500 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139503 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139506 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139509 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139512 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139514 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139518 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139521 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139523 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139526 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139529 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139532 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:15:12.141874 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139534 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139536 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139539 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139541 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139544 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139547 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139549 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139553 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139555 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139558 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139561 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139564 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139566 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139569 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139571 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139574 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139576 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139581 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139583 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139586 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:15:12.142336 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139589 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139591 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139593 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139596 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139599 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139601 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139605 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139608 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139611 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139614 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139617 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139620 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139623 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139626 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139628 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139631 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139634 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139637 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139639 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:15:12.142834 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139642 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139645 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.139648 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140021 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140026 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140028 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140031 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140034 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140037 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140040 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140042 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140045 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140048 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140050 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140053 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140056 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140059 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140061 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140064 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140067 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:15:12.143321 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140070 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140072 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140075 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140077 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140080 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140083 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140085 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140088 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140091 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140093 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140096 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140099 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140101 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140104 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140107 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140110 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140112 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140115 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140118 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140120 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:15:12.143827 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140123 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140126 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140128 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140131 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140133 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140135 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140138 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140141 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140143 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140146 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140149 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140152 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140155 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140157 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140160 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140164 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140167 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140171 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140173 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:15:12.144328 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140176 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140179 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140182 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140185 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140188 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140192 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140195 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140198 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140200 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140203 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140206 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140208 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140212 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140215 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140218 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140221 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140224 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140227 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140229 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:15:12.144804 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140232 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140236 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140239 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140241 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140244 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140246 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140249 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140252 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140254 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140256 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140259 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140261 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140266 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140269 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140271 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140274 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140276 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140279 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140281 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140284 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:15:12.145366 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140286 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140289 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140292 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.140294 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140364 2578 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140405 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140414 2578 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140418 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140422 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140426 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140429 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140433 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140437 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140440 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140443 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140446 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140449 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140453 2578 flags.go:64] FLAG: --cgroup-root="" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140455 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140459 2578 flags.go:64] FLAG: --client-ca-file="" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140462 2578 flags.go:64] FLAG: --cloud-provider="external" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140464 2578 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140469 2578 flags.go:64] FLAG: --cluster-domain="" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140472 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 14:15:12.145871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140477 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140480 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140484 2578 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140487 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140490 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140493 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140496 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140499 2578 flags.go:64] FLAG: --contention-profiling="false" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140503 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140506 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140512 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140515 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140519 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140522 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140525 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140528 2578 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140531 2578 flags.go:64] FLAG: --enable-server="true" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140537 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140541 2578 flags.go:64] FLAG: --event-burst="100" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140544 2578 flags.go:64] FLAG: --event-qps="50" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140547 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140551 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140554 2578 flags.go:64] FLAG: --eviction-hard="" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140557 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 14:15:12.146436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140560 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140563 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140566 2578 flags.go:64] FLAG: --eviction-soft="" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140569 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140571 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140574 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140577 2578 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140580 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140584 2578 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140587 2578 flags.go:64] FLAG: --feature-gates="" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140590 2578 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140593 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140596 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140599 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140602 2578 flags.go:64] FLAG: --healthz-port="10248" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140605 2578 flags.go:64] FLAG: --help="false" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140608 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140611 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140614 2578 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140618 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140621 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140625 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140627 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 14:15:12.147017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140630 2578 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140633 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140637 2578 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140640 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140643 2578 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140646 2578 flags.go:64] FLAG: --kube-reserved="" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140648 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140651 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140654 2578 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140657 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140660 2578 flags.go:64] FLAG: --lock-file="" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140663 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140666 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140668 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140674 2578 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140676 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140679 2578 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140683 2578 flags.go:64] FLAG: --logging-format="text" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140686 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140689 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140692 2578 flags.go:64] FLAG: --manifest-url="" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140695 2578 flags.go:64] FLAG: --manifest-url-header="" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140699 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140702 2578 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140706 2578 flags.go:64] FLAG: --max-pods="110" Sep 11 14:15:12.147614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140709 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140712 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140714 2578 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140717 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140721 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140724 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140753 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140761 2578 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140764 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140768 2578 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140771 2578 flags.go:64] FLAG: --pod-cidr="" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140774 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140780 2578 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140782 2578 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140786 2578 flags.go:64] FLAG: --pods-per-core="0" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140789 2578 flags.go:64] FLAG: --port="10250" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140792 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140795 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b8b3784caf7d5cea" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140798 2578 flags.go:64] FLAG: --qos-reserved="" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140801 2578 flags.go:64] FLAG: --read-only-port="10255" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140804 2578 flags.go:64] FLAG: --register-node="true" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140807 2578 flags.go:64] FLAG: --register-with-taints="" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140811 2578 flags.go:64] FLAG: --registry-burst="10" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140813 2578 flags.go:64] FLAG: --registry-qps="5" Sep 11 14:15:12.148617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140816 2578 flags.go:64] FLAG: --reserved-cpus="" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140820 2578 flags.go:64] FLAG: --reserved-memory="" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140824 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140827 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140830 2578 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140833 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140836 2578 flags.go:64] FLAG: --runonce="false" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140839 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140841 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140845 2578 flags.go:64] FLAG: --seccomp-default="false" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140847 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140850 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140853 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140857 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140860 2578 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140863 2578 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140866 2578 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140869 2578 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140873 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140876 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140879 2578 flags.go:64] FLAG: --system-cgroups="" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140882 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140887 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140890 2578 flags.go:64] FLAG: --tls-cert-file="" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140893 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 14:15:12.149240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140898 2578 flags.go:64] FLAG: --tls-min-version="" Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140901 2578 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140904 2578 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140906 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140909 2578 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140912 2578 flags.go:64] FLAG: --v="2" Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140917 2578 flags.go:64] FLAG: --version="false" Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140923 2578 flags.go:64] FLAG: --vmodule="" Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140928 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.140931 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141041 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141044 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141048 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141051 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141053 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141056 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141059 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141062 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141065 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141067 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141070 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141073 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:15:12.149881 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141075 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141078 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141081 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141085 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141087 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141090 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141093 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141096 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141098 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141101 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141103 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141106 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141109 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141111 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141114 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141117 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141119 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141122 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141126 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141129 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141132 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:15:12.150407 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141134 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141137 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141140 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141143 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141145 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141148 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141150 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141153 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141156 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141158 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141161 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141164 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141166 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141169 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141173 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141176 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141178 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141181 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141184 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:15:12.150927 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141187 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141189 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141192 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141194 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141197 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141200 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141203 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141205 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141208 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141211 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141214 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141217 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141220 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141222 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141225 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141227 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141230 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141234 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141238 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:15:12.151423 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141241 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141244 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141246 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141249 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141252 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141254 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141258 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141261 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141266 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141269 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141272 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141274 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141277 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141280 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141282 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141285 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141288 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141290 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141293 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141296 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:15:12.151926 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141298 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141301 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141303 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141307 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141310 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141313 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141317 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.141321 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.141780 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.149792 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.149806 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.150477 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.150485 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150541 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:15:12.152408 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150546 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150563 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150567 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150571 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150574 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150576 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150579 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150582 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150585 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150588 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150591 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150594 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150597 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150599 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150602 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150604 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150607 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150610 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150613 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150615 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:15:12.152882 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150618 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150620 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150625 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150629 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150632 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150635 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150639 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150643 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150648 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150651 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150654 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150657 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150660 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150662 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150665 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150668 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150671 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150673 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:15:12.153373 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150676 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150679 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150681 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150684 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150686 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150689 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150692 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150696 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150698 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150701 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150703 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150706 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150709 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150712 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150714 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150717 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150720 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150722 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150743 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150748 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:15:12.153825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150750 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150753 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150756 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150759 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150762 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150765 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150767 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150770 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150772 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150775 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150777 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150780 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150783 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150785 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150788 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150790 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150793 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150795 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150798 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150801 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:15:12.154304 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150805 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150807 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150810 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150813 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150815 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150818 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150821 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150823 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150825 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150828 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150830 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150833 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150835 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150838 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150841 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150843 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150846 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150849 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150852 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:15:12.154796 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150854 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.150859 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150975 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150980 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150983 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150986 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150989 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150992 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150995 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.150997 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151001 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151004 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151006 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151009 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151012 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:15:12.155255 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151015 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151018 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151020 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151023 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151026 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151028 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151031 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151033 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151036 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151039 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151041 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151044 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151046 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151049 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151052 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151055 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151057 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151060 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151063 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151066 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:15:12.155628 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151068 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151072 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151076 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151080 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151083 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151087 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151091 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151095 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151100 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151104 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151108 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151113 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151121 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151124 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151127 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151130 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151132 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151135 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151137 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:15:12.156161 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151140 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151142 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151145 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151148 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151150 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151153 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151156 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151158 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151161 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151164 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151166 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151169 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151172 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151174 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151177 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151179 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151182 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151185 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151187 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:15:12.156609 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151190 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151193 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151195 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151198 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151201 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151204 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151208 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151212 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151215 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151217 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151220 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151222 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151225 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151227 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151230 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151232 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151235 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151238 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151242 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:15:12.157129 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151245 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151247 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151250 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151252 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151255 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151258 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151262 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151265 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:12.151268 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.151272 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.151369 2578 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.153304 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.154044 2578 server.go:1019] "Starting client certificate rotation" Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.154143 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 14:15:12.157636 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.154183 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 14:15:12.158023 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.156094 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 14:15:12.158023 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.157186 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 14:15:12.166874 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.166856 2578 log.go:25] "Validated CRI v1 runtime API" Sep 11 14:15:12.169366 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.169352 2578 log.go:25] "Validated CRI v1 image API" Sep 11 14:15:12.170147 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.170133 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 14:15:12.175322 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.175303 2578 fs.go:135] Filesystem UUIDs: map[0d88ea05-9651-480b-a6d9-f8d781fda284:/dev/nvme0n1p3 65ff2f7a-951b-4a54-9a9f-03ce1679a412:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 11 14:15:12.175382 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.175321 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 14:15:12.182044 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.181938 2578 manager.go:216] Machine: {Timestamp:2026-09-11 14:15:12.180806797 +0000 UTC m=+0.266263148 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100501 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23d8d25c9c974612bc47178e5b490c SystemUUID:ec23d8d2-5c9c-9746-12bc-47178e5b490c BootID:ffcfeded-2cb4-4464-a4cb-72458f951462 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b4:e5:80:7f:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b4:e5:80:7f:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:d6:05:08:6c:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 14:15:12.182044 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.182037 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 14:15:12.182167 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.182139 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 14:15:12.184616 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.184589 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 14:15:12.184755 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.184620 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-57.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 14:15:12.185249 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.185239 2578 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 14:15:12.185285 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.185251 2578 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 14:15:12.185285 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.185263 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 14:15:12.185804 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.185794 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 14:15:12.186551 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.186542 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 14:15:12.189614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.189592 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 14:15:12.190799 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.190786 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 11 14:15:12.190912 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.190902 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 14:15:12.190954 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.190935 2578 kubelet.go:486] "Attempting to sync node with API server" Sep 11 14:15:12.190954 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.190947 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 14:15:12.191009 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.190962 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 14:15:12.191009 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.190975 2578 kubelet.go:398] "Adding apiserver pod source" Sep 11 14:15:12.191009 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.190983 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 14:15:12.191332 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.191320 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 14:15:12.191374 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.191340 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 14:15:12.193054 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.193036 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 14:15:12.194577 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.194564 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 14:15:12.194631 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.194593 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 14:15:12.196828 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.196815 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 14:15:12.196871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.196834 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 14:15:12.196871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.196841 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 14:15:12.196871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.196847 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 14:15:12.196871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.196855 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 14:15:12.196871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.196860 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 14:15:12.196871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.196874 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 14:15:12.197023 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.196879 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 14:15:12.197023 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.196887 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 14:15:12.197023 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.196892 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 14:15:12.197023 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.196901 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 14:15:12.197023 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.196906 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 14:15:12.197769 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.197755 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 14:15:12.197816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.197773 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 14:15:12.201609 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.201594 2578 server.go:1268] "Started kubelet" Sep 11 14:15:12.201703 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.201669 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 14:15:12.201824 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.201762 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 14:15:12.201878 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.201847 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 14:15:12.202378 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.202363 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-57.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 14:15:12.202896 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.202864 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 14:15:12.202896 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.202861 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 14:15:12.203200 ip-10-0-131-57 systemd[1]: Started Kubernetes Kubelet. Sep 11 14:15:12.204190 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.204174 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 14:15:12.205543 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.205524 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 14:15:12.205637 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.205625 2578 server.go:310] "Adding debug handlers to kubelet server" Sep 11 14:15:12.207926 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.207910 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 14:15:12.208255 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.208229 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 14:15:12.208375 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.208356 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:12.208881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.208774 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 14:15:12.209165 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.208392 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-57.ec2.internal.18d449d0969912e4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-57.ec2.internal,UID:ip-10-0-131-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-57.ec2.internal,},FirstTimestamp:2026-09-11 14:15:12.201568996 +0000 UTC m=+0.287025347,LastTimestamp:2026-09-11 14:15:12.201568996 +0000 UTC m=+0.287025347,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-57.ec2.internal,}" Sep 11 14:15:12.209510 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.209440 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 14:15:12.209510 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.209456 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 14:15:12.209630 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.209543 2578 reconstruct.go:97] "Volume reconstruction finished" Sep 11 14:15:12.209630 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.209553 2578 reconciler.go:29] "Reconciler: start to sync state" Sep 11 14:15:12.210279 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.210135 2578 factory.go:55] Registering systemd factory Sep 11 14:15:12.210279 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.210150 2578 factory.go:223] Registration of the systemd container factory successfully Sep 11 14:15:12.210406 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.210390 2578 factory.go:153] Registering CRI-O factory Sep 11 14:15:12.210443 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.210411 2578 factory.go:223] Registration of the crio container factory successfully Sep 11 14:15:12.210640 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.210625 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 14:15:12.210696 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.210656 2578 factory.go:103] Registering Raw factory Sep 11 14:15:12.210696 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.210676 2578 manager.go:1196] Started watching for new ooms in manager Sep 11 14:15:12.211811 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.211795 2578 manager.go:349] Starting recovery of all containers Sep 11 14:15:12.213873 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.213844 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 14:15:12.213972 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.213856 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 14:15:12.223765 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.223747 2578 manager.go:354] Recovery completed Sep 11 14:15:12.225947 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.225928 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h4tsl" Sep 11 14:15:12.228080 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.228067 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:15:12.229712 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.229697 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:15:12.229795 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.229723 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:15:12.229795 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.229778 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:15:12.230276 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.230263 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 14:15:12.230276 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.230274 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 14:15:12.230388 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.230291 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 11 14:15:12.230388 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.230307 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:15:12.230977 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.230960 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:15:12.231034 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.230988 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:15:12.231034 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.230998 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:15:12.232062 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.231982 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-57.ec2.internal.18d449d098467982 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-57.ec2.internal,UID:ip-10-0-131-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-57.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-57.ec2.internal,},FirstTimestamp:2026-09-11 14:15:12.22971021 +0000 UTC m=+0.315166561,LastTimestamp:2026-09-11 14:15:12.22971021 +0000 UTC m=+0.315166561,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-57.ec2.internal,}" Sep 11 14:15:12.232461 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.232441 2578 policy_none.go:49] "None policy: Start" Sep 11 14:15:12.232497 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.232464 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 14:15:12.232497 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.232475 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 14:15:12.234374 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.234362 2578 policy_none.go:47] "Start" Sep 11 14:15:12.235662 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.235645 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h4tsl" Sep 11 14:15:12.242883 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.242824 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-57.ec2.internal.18d449d09847402a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-57.ec2.internal,UID:ip-10-0-131-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-57.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-57.ec2.internal,},FirstTimestamp:2026-09-11 14:15:12.229761066 +0000 UTC m=+0.315217422,LastTimestamp:2026-09-11 14:15:12.229761066 +0000 UTC m=+0.315217422,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-57.ec2.internal,}" Sep 11 14:15:12.251834 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.251776 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-57.ec2.internal.18d449d0984797af default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-57.ec2.internal,UID:ip-10-0-131-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-57.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-57.ec2.internal,},FirstTimestamp:2026-09-11 14:15:12.229783471 +0000 UTC m=+0.315239824,LastTimestamp:2026-09-11 14:15:12.229783471 +0000 UTC m=+0.315239824,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-57.ec2.internal,}" Sep 11 14:15:12.261704 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.261637 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-57.ec2.internal.18d449d098467982\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-57.ec2.internal.18d449d098467982 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-57.ec2.internal,UID:ip-10-0-131-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-57.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-57.ec2.internal,},FirstTimestamp:2026-09-11 14:15:12.22971021 +0000 UTC m=+0.315166561,LastTimestamp:2026-09-11 14:15:12.230974965 +0000 UTC m=+0.316431320,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-57.ec2.internal,}" Sep 11 14:15:12.265358 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.265345 2578 manager.go:337] "Starting Device Plugin manager" Sep 11 14:15:12.265443 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.265381 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 14:15:12.265443 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.265392 2578 server.go:87] "Starting device plugin registration server" Sep 11 14:15:12.265630 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.265616 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 14:15:12.265707 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.265633 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 14:15:12.265781 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.265719 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 14:15:12.265886 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.265874 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 14:15:12.265972 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.265887 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 14:15:12.266294 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.266280 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 14:15:12.266361 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.266317 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:12.274361 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.274307 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-57.ec2.internal.18d449d09847402a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-57.ec2.internal.18d449d09847402a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-57.ec2.internal,UID:ip-10-0-131-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-57.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-57.ec2.internal,},FirstTimestamp:2026-09-11 14:15:12.229761066 +0000 UTC m=+0.315217422,LastTimestamp:2026-09-11 14:15:12.230993408 +0000 UTC m=+0.316449759,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-57.ec2.internal,}" Sep 11 14:15:12.281921 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.281904 2578 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2607 into the system container \"/system.slice\": " Sep 11 14:15:12.283755 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.283680 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-57.ec2.internal.18d449d0984797af\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-57.ec2.internal.18d449d0984797af default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-57.ec2.internal,UID:ip-10-0-131-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-57.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-57.ec2.internal,},FirstTimestamp:2026-09-11 14:15:12.229783471 +0000 UTC m=+0.315239824,LastTimestamp:2026-09-11 14:15:12.231001552 +0000 UTC m=+0.316457903,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-57.ec2.internal,}" Sep 11 14:15:12.293387 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.293328 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-57.ec2.internal.18d449d09b1a4416 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-57.ec2.internal,UID:ip-10-0-131-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-131-57.ec2.internal,},FirstTimestamp:2026-09-11 14:15:12.277144598 +0000 UTC m=+0.362600942,LastTimestamp:2026-09-11 14:15:12.277144598 +0000 UTC m=+0.362600942,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-57.ec2.internal,}" Sep 11 14:15:12.366688 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.366666 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:15:12.367470 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.367449 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:15:12.367546 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.367482 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:15:12.367546 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.367494 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:15:12.367546 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.367517 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.376568 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.376507 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-57.ec2.internal.18d449d098467982\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-57.ec2.internal.18d449d098467982 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-57.ec2.internal,UID:ip-10-0-131-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-57.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-57.ec2.internal,},FirstTimestamp:2026-09-11 14:15:12.22971021 +0000 UTC m=+0.315166561,LastTimestamp:2026-09-11 14:15:12.367469035 +0000 UTC m=+0.452925372,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-57.ec2.internal,}" Sep 11 14:15:12.382158 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.382137 2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-57.ec2.internal\" not found" node="ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.421354 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.421335 2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-57.ec2.internal\" not found" node="ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.456199 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.456171 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 14:15:12.457328 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.457313 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 14:15:12.457382 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.457358 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 14:15:12.457382 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.457378 2578 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 14:15:12.457452 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.457411 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 14:15:12.460497 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.460476 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:15:12.558620 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.558001 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal"] Sep 11 14:15:12.558717 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.558657 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:15:12.559600 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.559582 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:15:12.559649 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.559613 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:15:12.559649 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.559623 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:15:12.561973 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.561961 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:15:12.562084 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.562071 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.562124 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.562099 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:15:12.562630 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.562617 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:15:12.562630 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.562626 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:15:12.562744 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.562644 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:15:12.562744 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.562644 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:15:12.562744 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.562654 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:15:12.562744 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.562686 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:15:12.564887 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.564873 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.564963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.564897 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:15:12.565540 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.565524 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:15:12.565602 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.565553 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:15:12.565602 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.565568 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:15:12.582960 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.582926 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:15:12.584171 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.584147 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:15:12.584254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.584174 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:15:12.584254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.584187 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:15:12.584254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.584205 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.592305 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.592285 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.592305 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.592304 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-57.ec2.internal\": node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:12.593363 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.593350 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-57.ec2.internal\" not found" node="ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.597876 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.597860 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-57.ec2.internal\" not found" node="ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.612291 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.612272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d29ff600cf421cf8453088acb7f44e53-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal\" (UID: \"d29ff600cf421cf8453088acb7f44e53\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.612384 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.612327 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d29ff600cf421cf8453088acb7f44e53-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal\" (UID: \"d29ff600cf421cf8453088acb7f44e53\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.612384 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.612349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ffbb18c601b83060fb6937c4ad273ac5-config\") pod \"kube-apiserver-proxy-ip-10-0-131-57.ec2.internal\" (UID: \"ffbb18c601b83060fb6937c4ad273ac5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.617679 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.617666 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:12.713174 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.713131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d29ff600cf421cf8453088acb7f44e53-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal\" (UID: \"d29ff600cf421cf8453088acb7f44e53\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.713174 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.713154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d29ff600cf421cf8453088acb7f44e53-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal\" (UID: \"d29ff600cf421cf8453088acb7f44e53\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.713174 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.713172 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ffbb18c601b83060fb6937c4ad273ac5-config\") pod \"kube-apiserver-proxy-ip-10-0-131-57.ec2.internal\" (UID: \"ffbb18c601b83060fb6937c4ad273ac5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.713307 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.713228 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ffbb18c601b83060fb6937c4ad273ac5-config\") pod \"kube-apiserver-proxy-ip-10-0-131-57.ec2.internal\" (UID: \"ffbb18c601b83060fb6937c4ad273ac5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.713307 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.713236 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d29ff600cf421cf8453088acb7f44e53-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal\" (UID: \"d29ff600cf421cf8453088acb7f44e53\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.713307 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.713236 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d29ff600cf421cf8453088acb7f44e53-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal\" (UID: \"d29ff600cf421cf8453088acb7f44e53\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.717751 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.717739 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:12.818653 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.818623 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:12.895830 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.895798 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.901540 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:12.901525 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal" Sep 11 14:15:12.919865 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:12.919696 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:13.020238 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:13.020162 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:13.111305 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.111286 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:15:13.120294 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:13.120277 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:13.154511 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.154481 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 14:15:13.154940 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.154625 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 14:15:13.154940 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.154651 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 14:15:13.221354 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:13.221328 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:13.298215 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.298127 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:15:13.299981 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.299949 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:15:13.300079 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.299990 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:15:13.300079 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.300006 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:15:13.321414 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:13.321388 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:13.342280 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.342259 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:15:13.379507 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.379477 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 14:10:12 +0000 UTC" deadline="2028-02-20 13:26:51.136543584 +0000 UTC" Sep 11 14:15:13.379507 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.379505 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12647h11m37.757041912s" Sep 11 14:15:13.421737 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:13.421711 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:13.522414 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:13.522388 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:13.595954 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:13.595926 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29ff600cf421cf8453088acb7f44e53.slice/crio-0d5395547c7d16c255cf7a38f30fd88934f0dc4c85ebeb9a5793aa1a23b4bd55 WatchSource:0}: Error finding container 0d5395547c7d16c255cf7a38f30fd88934f0dc4c85ebeb9a5793aa1a23b4bd55: Status 404 returned error can't find the container with id 0d5395547c7d16c255cf7a38f30fd88934f0dc4c85ebeb9a5793aa1a23b4bd55 Sep 11 14:15:13.596576 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:13.596545 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbb18c601b83060fb6937c4ad273ac5.slice/crio-c1e347ea5724e1e03e84068c0ca07f3ffdf1fac30e9946e031ed68ac66a42500 WatchSource:0}: Error finding container c1e347ea5724e1e03e84068c0ca07f3ffdf1fac30e9946e031ed68ac66a42500: Status 404 returned error can't find the container with id c1e347ea5724e1e03e84068c0ca07f3ffdf1fac30e9946e031ed68ac66a42500 Sep 11 14:15:13.601996 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.601981 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:15:13.622699 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:13.622677 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-57.ec2.internal\" not found" Sep 11 14:15:13.623542 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.623526 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:15:13.709822 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.709785 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" Sep 11 14:15:13.724589 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.724570 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 14:15:13.725705 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.725692 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal" Sep 11 14:15:13.737622 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:13.737605 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 14:15:14.194826 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.194797 2578 apiserver.go:52] "Watching apiserver" Sep 11 14:15:14.206311 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.205911 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 14:15:14.206516 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.206455 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal","openshift-multus/network-metrics-daemon-pp2sq","openshift-network-diagnostics/network-check-target-75q2d","openshift-network-operator/iptables-alerter-wpjnm","openshift-ovn-kubernetes/ovnkube-node-8k54l","kube-system/global-pull-secret-syncer-crnx9","kube-system/konnectivity-agent-vszlk","kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx","openshift-cluster-node-tuning-operator/tuned-b78k9","openshift-dns/node-resolver-2fj2j","openshift-multus/multus-6pfsh","openshift-multus/multus-additional-cni-plugins-rvgbl","openshift-image-registry/node-ca-7plbw"] Sep 11 14:15:14.209050 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.208862 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 14:15:14.209175 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.209130 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vszlk" Sep 11 14:15:14.212068 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.212042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.213070 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.213047 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 14:15:14.213192 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.213158 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vkn2t\"" Sep 11 14:15:14.215180 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.215155 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 14:15:14.215296 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.215281 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 14:15:14.215552 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.215507 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 14:15:14.216070 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.216046 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 14:15:14.216364 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.216344 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 14:15:14.216647 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.216628 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hnk2c\"" Sep 11 14:15:14.216983 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.216963 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 14:15:14.217222 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.217198 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:14.217340 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.217306 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:14.218782 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.218531 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wpjnm" Sep 11 14:15:14.222113 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.221523 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:15:14.224168 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.223310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1994440-b56e-4b0c-b414-f68944e12f56-agent-certs\") pod \"konnectivity-agent-vszlk\" (UID: \"c1994440-b56e-4b0c-b414-f68944e12f56\") " pod="kube-system/konnectivity-agent-vszlk" Sep 11 14:15:14.224168 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.223346 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1994440-b56e-4b0c-b414-f68944e12f56-konnectivity-ca\") pod \"konnectivity-agent-vszlk\" (UID: \"c1994440-b56e-4b0c-b414-f68944e12f56\") " pod="kube-system/konnectivity-agent-vszlk" Sep 11 14:15:14.224168 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.223437 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 14:15:14.224168 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.223698 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 14:15:14.224168 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.223938 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5tx9z\"" Sep 11 14:15:14.224460 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.224236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.226687 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.226566 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:14.226687 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.226639 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:14.227315 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.227050 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 14:15:14.227315 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.227080 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-65s69\"" Sep 11 14:15:14.228645 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.228543 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 14:15:14.228645 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.228566 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 14:15:14.228838 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.228756 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 14:15:14.229016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.228927 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 14:15:14.229016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.228952 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 14:15:14.229399 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.229381 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.233382 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.232102 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7plbw" Sep 11 14:15:14.233382 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.233080 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 14:15:14.233382 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.233326 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 14:15:14.233585 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.233444 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nwt69\"" Sep 11 14:15:14.233585 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.233568 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 14:15:14.233767 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.233712 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 14:15:14.235060 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.234796 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 14:15:14.235384 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.235370 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:14.235552 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.235532 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:14.237357 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.236736 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 14:15:14.237357 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.237107 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 14:15:14.237357 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.237347 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4fxn9\"" Sep 11 14:15:14.238160 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.237724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.240538 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.240520 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:15:14.240635 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.240598 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 14:15:14.240691 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.240631 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9m29t\"" Sep 11 14:15:14.240837 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.240819 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2fj2j" Sep 11 14:15:14.244013 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.243903 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 14:15:14.244269 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.244250 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 14:15:14.245164 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.244977 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-npqv2\"" Sep 11 14:15:14.247503 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.247483 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.251623 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.251454 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m548t\"" Sep 11 14:15:14.251787 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.251765 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 14:15:14.263517 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.263487 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gjb78" Sep 11 14:15:14.277344 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.277319 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gjb78" Sep 11 14:15:14.310914 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.310894 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 14:15:14.323570 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.323539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:14.323666 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.323580 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-run-systemd\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.323666 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.323609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-run-openvswitch\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.323666 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.323651 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-systemd\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.323797 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.323697 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49b6af32-4870-47da-98f8-3fb48b1a63df-cni-binary-copy\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.323797 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.323750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dac8086f-f464-4481-88bb-ada601e7351d-multus-daemon-config\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.323797 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.323779 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-run\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.323893 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.323804 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-tuned\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.323893 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.323836 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a062adf8-6207-40d7-8eb0-302fede4913a-tmp-dir\") pod \"node-resolver-2fj2j\" (UID: \"a062adf8-6207-40d7-8eb0-302fede4913a\") " pod="openshift-dns/node-resolver-2fj2j" Sep 11 14:15:14.323893 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.323878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1994440-b56e-4b0c-b414-f68944e12f56-konnectivity-ca\") pod \"konnectivity-agent-vszlk\" (UID: \"c1994440-b56e-4b0c-b414-f68944e12f56\") " pod="kube-system/konnectivity-agent-vszlk" Sep 11 14:15:14.324035 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.323906 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dac8086f-f464-4481-88bb-ada601e7351d-cni-binary-copy\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.324035 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.323931 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-log-socket\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.324035 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324013 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-run-ovn-kubernetes\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.324225 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-kubernetes\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.324225 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:14.324225 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.324225 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-sys-fs\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.324225 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-cni-netd\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.324225 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324201 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-var-lib-kubelet\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.324525 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49b6af32-4870-47da-98f8-3fb48b1a63df-cnibin\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.324525 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49b6af32-4870-47da-98f8-3fb48b1a63df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.324525 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8202348f-3147-4b1e-bbd1-c476ac7260fb-iptables-alerter-script\") pod \"iptables-alerter-wpjnm\" (UID: \"8202348f-3147-4b1e-bbd1-c476ac7260fb\") " pod="openshift-network-operator/iptables-alerter-wpjnm" Sep 11 14:15:14.324525 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-run-netns\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.324525 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-sysctl-d\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.324525 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49b6af32-4870-47da-98f8-3fb48b1a63df-system-cni-dir\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.324525 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324462 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1994440-b56e-4b0c-b414-f68944e12f56-konnectivity-ca\") pod \"konnectivity-agent-vszlk\" (UID: \"c1994440-b56e-4b0c-b414-f68944e12f56\") " pod="kube-system/konnectivity-agent-vszlk" Sep 11 14:15:14.324525 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-dbus\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324551 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-system-cni-dir\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324580 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-run-k8s-cni-cncf-io\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbj7f\" (UniqueName: \"kubernetes.io/projected/dac8086f-f464-4481-88bb-ada601e7351d-kube-api-access-cbj7f\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-etc-openvswitch\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-multus-socket-dir-parent\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-ovn-node-metrics-cert\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324704 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4dec9ea-724d-422e-8783-4795fe390f80-host\") pod \"node-ca-7plbw\" (UID: \"d4dec9ea-724d-422e-8783-4795fe390f80\") " pod="openshift-image-registry/node-ca-7plbw" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324727 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-sysconfig\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-sysctl-conf\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vrfd\" (UniqueName: \"kubernetes.io/projected/a062adf8-6207-40d7-8eb0-302fede4913a-kube-api-access-7vrfd\") pod \"node-resolver-2fj2j\" (UID: \"a062adf8-6207-40d7-8eb0-302fede4913a\") " pod="openshift-dns/node-resolver-2fj2j" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-socket-dir\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnjvq\" (UniqueName: \"kubernetes.io/projected/df1160f9-2ec0-4907-9ca2-4b269875d943-kube-api-access-lnjvq\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324870 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-var-lib-kubelet\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-etc-kubernetes\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.324922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-env-overrides\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.325719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324938 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-ovnkube-script-lib\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.325719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.324976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb2tn\" (UniqueName: \"kubernetes.io/projected/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-kube-api-access-tb2tn\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.325719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-lib-modules\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.325719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49b6af32-4870-47da-98f8-3fb48b1a63df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.325719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325078 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-var-lib-cni-bin\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.325719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325104 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-multus-conf-dir\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.325719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-ovnkube-config\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.325719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325161 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-registration-dir\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.325719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-etc-selinux\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.325719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8202348f-3147-4b1e-bbd1-c476ac7260fb-host-slash\") pod \"iptables-alerter-wpjnm\" (UID: \"8202348f-3147-4b1e-bbd1-c476ac7260fb\") " pod="openshift-network-operator/iptables-alerter-wpjnm" Sep 11 14:15:14.325719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-node-log\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.325719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-sys\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.325719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325324 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-host\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/230f41e9-12d5-4191-9eef-f5af5c5090f8-tmp\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325870 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx66x\" (UniqueName: \"kubernetes.io/projected/5647f838-aa18-4f4e-9d6d-522d71950a7c-kube-api-access-tx66x\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-cnibin\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-run-netns\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlws8\" (UniqueName: \"kubernetes.io/projected/8202348f-3147-4b1e-bbd1-c476ac7260fb-kube-api-access-nlws8\") pod \"iptables-alerter-wpjnm\" (UID: \"8202348f-3147-4b1e-bbd1-c476ac7260fb\") " pod="openshift-network-operator/iptables-alerter-wpjnm" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325973 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8l2\" (UniqueName: \"kubernetes.io/projected/d4dec9ea-724d-422e-8783-4795fe390f80-kube-api-access-8g8l2\") pod \"node-ca-7plbw\" (UID: \"d4dec9ea-724d-422e-8783-4795fe390f80\") " pod="openshift-image-registry/node-ca-7plbw" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.325998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49b6af32-4870-47da-98f8-3fb48b1a63df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326024 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72nr9\" (UniqueName: \"kubernetes.io/projected/49b6af32-4870-47da-98f8-3fb48b1a63df-kube-api-access-72nr9\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326048 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh7cj\" (UniqueName: \"kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj\") pod \"network-check-target-75q2d\" (UID: \"c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f\") " pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-device-dir\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-run-multus-certs\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-run-ovn\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326138 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a062adf8-6207-40d7-8eb0-302fede4913a-hosts-file\") pod \"node-resolver-2fj2j\" (UID: \"a062adf8-6207-40d7-8eb0-302fede4913a\") " pod="openshift-dns/node-resolver-2fj2j" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-kubelet-config\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1994440-b56e-4b0c-b414-f68944e12f56-agent-certs\") pod \"konnectivity-agent-vszlk\" (UID: \"c1994440-b56e-4b0c-b414-f68944e12f56\") " pod="kube-system/konnectivity-agent-vszlk" Sep 11 14:15:14.326391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-multus-cni-dir\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.327205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-var-lib-cni-multus\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.327205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-hostroot\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.327205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326286 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-kubelet\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.327205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-cni-bin\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.327205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.327205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-slash\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.327205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326440 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4dec9ea-724d-422e-8783-4795fe390f80-serviceca\") pod \"node-ca-7plbw\" (UID: \"d4dec9ea-724d-422e-8783-4795fe390f80\") " pod="openshift-image-registry/node-ca-7plbw" Sep 11 14:15:14.327205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326462 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-modprobe-d\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.327205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326485 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zffht\" (UniqueName: \"kubernetes.io/projected/230f41e9-12d5-4191-9eef-f5af5c5090f8-kube-api-access-zffht\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.327205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-systemd-units\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.327205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-var-lib-openvswitch\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.327205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.326721 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 14:15:14.331482 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.331451 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1994440-b56e-4b0c-b414-f68944e12f56-agent-certs\") pod \"konnectivity-agent-vszlk\" (UID: \"c1994440-b56e-4b0c-b414-f68944e12f56\") " pod="kube-system/konnectivity-agent-vszlk" Sep 11 14:15:14.426979 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.426951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-multus-socket-dir-parent\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.427152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.426991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-ovn-node-metrics-cert\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.427152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4dec9ea-724d-422e-8783-4795fe390f80-host\") pod \"node-ca-7plbw\" (UID: \"d4dec9ea-724d-422e-8783-4795fe390f80\") " pod="openshift-image-registry/node-ca-7plbw" Sep 11 14:15:14.427152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-sysconfig\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.427152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-sysctl-conf\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.427152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vrfd\" (UniqueName: \"kubernetes.io/projected/a062adf8-6207-40d7-8eb0-302fede4913a-kube-api-access-7vrfd\") pod \"node-resolver-2fj2j\" (UID: \"a062adf8-6207-40d7-8eb0-302fede4913a\") " pod="openshift-dns/node-resolver-2fj2j" Sep 11 14:15:14.427152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-socket-dir\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.427152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnjvq\" (UniqueName: \"kubernetes.io/projected/df1160f9-2ec0-4907-9ca2-4b269875d943-kube-api-access-lnjvq\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:14.427536 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-var-lib-kubelet\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.427536 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427171 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-multus-socket-dir-parent\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.427536 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-etc-kubernetes\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.427536 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-env-overrides\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.427536 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427244 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-var-lib-kubelet\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.427536 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427182 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4dec9ea-724d-422e-8783-4795fe390f80-host\") pod \"node-ca-7plbw\" (UID: \"d4dec9ea-724d-422e-8783-4795fe390f80\") " pod="openshift-image-registry/node-ca-7plbw" Sep 11 14:15:14.427536 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-ovnkube-script-lib\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.427536 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427383 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-socket-dir\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.427536 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427402 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-etc-kubernetes\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.427536 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb2tn\" (UniqueName: \"kubernetes.io/projected/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-kube-api-access-tb2tn\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427564 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-sysctl-conf\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427616 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-sysconfig\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-lib-modules\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49b6af32-4870-47da-98f8-3fb48b1a63df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-var-lib-cni-bin\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-multus-conf-dir\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-ovnkube-config\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427848 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-lib-modules\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-registration-dir\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-etc-selinux\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427927 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-registration-dir\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427962 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-env-overrides\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.427940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8202348f-3147-4b1e-bbd1-c476ac7260fb-host-slash\") pod \"iptables-alerter-wpjnm\" (UID: \"8202348f-3147-4b1e-bbd1-c476ac7260fb\") " pod="openshift-network-operator/iptables-alerter-wpjnm" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428026 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-node-log\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428038 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-multus-conf-dir\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428059 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-ovnkube-script-lib\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.428153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-sys\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428091 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-node-log\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-host\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/230f41e9-12d5-4191-9eef-f5af5c5090f8-tmp\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428158 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-var-lib-cni-bin\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428161 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx66x\" (UniqueName: \"kubernetes.io/projected/5647f838-aa18-4f4e-9d6d-522d71950a7c-kube-api-access-tx66x\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-cnibin\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-run-netns\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428280 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-host\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428320 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-etc-selinux\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428329 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-run-netns\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlws8\" (UniqueName: \"kubernetes.io/projected/8202348f-3147-4b1e-bbd1-c476ac7260fb-kube-api-access-nlws8\") pod \"iptables-alerter-wpjnm\" (UID: \"8202348f-3147-4b1e-bbd1-c476ac7260fb\") " pod="openshift-network-operator/iptables-alerter-wpjnm" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428377 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-cnibin\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8l2\" (UniqueName: \"kubernetes.io/projected/d4dec9ea-724d-422e-8783-4795fe390f80-kube-api-access-8g8l2\") pod \"node-ca-7plbw\" (UID: \"d4dec9ea-724d-422e-8783-4795fe390f80\") " pod="openshift-image-registry/node-ca-7plbw" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49b6af32-4870-47da-98f8-3fb48b1a63df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72nr9\" (UniqueName: \"kubernetes.io/projected/49b6af32-4870-47da-98f8-3fb48b1a63df-kube-api-access-72nr9\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428464 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-sys\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7cj\" (UniqueName: \"kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj\") pod \"network-check-target-75q2d\" (UID: \"c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f\") " pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:14.429265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428414 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8202348f-3147-4b1e-bbd1-c476ac7260fb-host-slash\") pod \"iptables-alerter-wpjnm\" (UID: \"8202348f-3147-4b1e-bbd1-c476ac7260fb\") " pod="openshift-network-operator/iptables-alerter-wpjnm" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-device-dir\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428514 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49b6af32-4870-47da-98f8-3fb48b1a63df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-run-multus-certs\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428541 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-ovnkube-config\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428563 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-run-multus-certs\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-run-ovn\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428612 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-device-dir\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a062adf8-6207-40d7-8eb0-302fede4913a-hosts-file\") pod \"node-resolver-2fj2j\" (UID: \"a062adf8-6207-40d7-8eb0-302fede4913a\") " pod="openshift-dns/node-resolver-2fj2j" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-kubelet-config\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-multus-cni-dir\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-var-lib-cni-multus\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428743 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-var-lib-cni-multus\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428778 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-kubelet-config\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428806 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-run-ovn\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428854 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-hostroot\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428870 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a062adf8-6207-40d7-8eb0-302fede4913a-hosts-file\") pod \"node-resolver-2fj2j\" (UID: \"a062adf8-6207-40d7-8eb0-302fede4913a\") " pod="openshift-dns/node-resolver-2fj2j" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428897 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-multus-cni-dir\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.430220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428913 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-hostroot\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.428988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-kubelet\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429019 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-cni-bin\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429022 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49b6af32-4870-47da-98f8-3fb48b1a63df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429045 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-slash\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429090 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-cni-bin\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429094 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-kubelet\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4dec9ea-724d-422e-8783-4795fe390f80-serviceca\") pod \"node-ca-7plbw\" (UID: \"d4dec9ea-724d-422e-8783-4795fe390f80\") " pod="openshift-image-registry/node-ca-7plbw" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429136 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429153 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-slash\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-modprobe-d\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zffht\" (UniqueName: \"kubernetes.io/projected/230f41e9-12d5-4191-9eef-f5af5c5090f8-kube-api-access-zffht\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-systemd-units\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-var-lib-openvswitch\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429313 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-modprobe-d\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.431015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-run-systemd\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-run-openvswitch\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429366 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-systemd-units\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-systemd\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49b6af32-4870-47da-98f8-3fb48b1a63df-cni-binary-copy\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dac8086f-f464-4481-88bb-ada601e7351d-multus-daemon-config\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-run\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429485 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-tuned\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a062adf8-6207-40d7-8eb0-302fede4913a-tmp-dir\") pod \"node-resolver-2fj2j\" (UID: \"a062adf8-6207-40d7-8eb0-302fede4913a\") " pod="openshift-dns/node-resolver-2fj2j" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429524 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d4dec9ea-724d-422e-8783-4795fe390f80-serviceca\") pod \"node-ca-7plbw\" (UID: \"d4dec9ea-724d-422e-8783-4795fe390f80\") " pod="openshift-image-registry/node-ca-7plbw" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429537 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dac8086f-f464-4481-88bb-ada601e7351d-cni-binary-copy\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429539 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-var-lib-openvswitch\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-log-socket\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429586 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-run-openvswitch\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-run-ovn-kubernetes\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429606 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-systemd\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-kubernetes\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429644 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-run\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.431963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.429732 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-kubernetes\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430053 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a062adf8-6207-40d7-8eb0-302fede4913a-tmp-dir\") pod \"node-resolver-2fj2j\" (UID: \"a062adf8-6207-40d7-8eb0-302fede4913a\") " pod="openshift-dns/node-resolver-2fj2j" Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430085 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-log-socket\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430120 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49b6af32-4870-47da-98f8-3fb48b1a63df-cni-binary-copy\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430132 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-run-ovn-kubernetes\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430135 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dac8086f-f464-4481-88bb-ada601e7351d-multus-daemon-config\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430164 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430194 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430196 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-run-systemd\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-sys-fs\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.430229 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.430271 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.430307 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs podName:df1160f9-2ec0-4907-9ca2-4b269875d943 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:14.930281898 +0000 UTC m=+3.015738268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs") pod "network-metrics-daemon-pp2sq" (UID: "df1160f9-2ec0-4907-9ca2-4b269875d943") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430272 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-sys-fs\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430328 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5647f838-aa18-4f4e-9d6d-522d71950a7c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.430332 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret podName:b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:14.93031647 +0000 UTC m=+3.015772817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret") pod "global-pull-secret-syncer-crnx9" (UID: "b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-cni-netd\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.432697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-var-lib-kubelet\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49b6af32-4870-47da-98f8-3fb48b1a63df-cnibin\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430432 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49b6af32-4870-47da-98f8-3fb48b1a63df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8202348f-3147-4b1e-bbd1-c476ac7260fb-iptables-alerter-script\") pod \"iptables-alerter-wpjnm\" (UID: \"8202348f-3147-4b1e-bbd1-c476ac7260fb\") " pod="openshift-network-operator/iptables-alerter-wpjnm" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-run-netns\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-sysctl-d\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430529 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-var-lib-kubelet\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49b6af32-4870-47da-98f8-3fb48b1a63df-system-cni-dir\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-dbus\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430570 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-cni-netd\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430586 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-system-cni-dir\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-run-k8s-cni-cncf-io\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430616 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-host-run-netns\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbj7f\" (UniqueName: \"kubernetes.io/projected/dac8086f-f464-4481-88bb-ada601e7351d-kube-api-access-cbj7f\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430656 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49b6af32-4870-47da-98f8-3fb48b1a63df-cnibin\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430661 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dac8086f-f464-4481-88bb-ada601e7351d-cni-binary-copy\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-etc-openvswitch\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.433560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430723 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-host-run-k8s-cni-cncf-io\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.434206 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430727 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dac8086f-f464-4481-88bb-ada601e7351d-system-cni-dir\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.434206 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430762 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-etc-openvswitch\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.434206 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430783 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49b6af32-4870-47da-98f8-3fb48b1a63df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.434206 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430833 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49b6af32-4870-47da-98f8-3fb48b1a63df-system-cni-dir\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.434206 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.430993 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-dbus\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:14.434206 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.431002 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-sysctl-d\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.434206 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.431152 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-ovn-node-metrics-cert\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.434206 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.431259 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/230f41e9-12d5-4191-9eef-f5af5c5090f8-tmp\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.434206 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.431275 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8202348f-3147-4b1e-bbd1-c476ac7260fb-iptables-alerter-script\") pod \"iptables-alerter-wpjnm\" (UID: \"8202348f-3147-4b1e-bbd1-c476ac7260fb\") " pod="openshift-network-operator/iptables-alerter-wpjnm" Sep 11 14:15:14.434206 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.433355 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/230f41e9-12d5-4191-9eef-f5af5c5090f8-etc-tuned\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.441654 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.441630 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:15:14.443116 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.443087 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vrfd\" (UniqueName: \"kubernetes.io/projected/a062adf8-6207-40d7-8eb0-302fede4913a-kube-api-access-7vrfd\") pod \"node-resolver-2fj2j\" (UID: \"a062adf8-6207-40d7-8eb0-302fede4913a\") " pod="openshift-dns/node-resolver-2fj2j" Sep 11 14:15:14.444570 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.444545 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zffht\" (UniqueName: \"kubernetes.io/projected/230f41e9-12d5-4191-9eef-f5af5c5090f8-kube-api-access-zffht\") pod \"tuned-b78k9\" (UID: \"230f41e9-12d5-4191-9eef-f5af5c5090f8\") " pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.444991 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.444894 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnjvq\" (UniqueName: \"kubernetes.io/projected/df1160f9-2ec0-4907-9ca2-4b269875d943-kube-api-access-lnjvq\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:14.444991 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.444966 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbj7f\" (UniqueName: \"kubernetes.io/projected/dac8086f-f464-4481-88bb-ada601e7351d-kube-api-access-cbj7f\") pod \"multus-6pfsh\" (UID: \"dac8086f-f464-4481-88bb-ada601e7351d\") " pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.446820 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.446775 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:14.446820 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.446801 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:14.446820 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.446814 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wh7cj for pod openshift-network-diagnostics/network-check-target-75q2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:14.447029 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.446876 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj podName:c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:14.946857808 +0000 UTC m=+3.032314169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wh7cj" (UniqueName: "kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj") pod "network-check-target-75q2d" (UID: "c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:14.450205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.450177 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx66x\" (UniqueName: \"kubernetes.io/projected/5647f838-aa18-4f4e-9d6d-522d71950a7c-kube-api-access-tx66x\") pod \"aws-ebs-csi-driver-node-vnwqx\" (UID: \"5647f838-aa18-4f4e-9d6d-522d71950a7c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.451172 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.451137 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8l2\" (UniqueName: \"kubernetes.io/projected/d4dec9ea-724d-422e-8783-4795fe390f80-kube-api-access-8g8l2\") pod \"node-ca-7plbw\" (UID: \"d4dec9ea-724d-422e-8783-4795fe390f80\") " pod="openshift-image-registry/node-ca-7plbw" Sep 11 14:15:14.451286 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.451263 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72nr9\" (UniqueName: \"kubernetes.io/projected/49b6af32-4870-47da-98f8-3fb48b1a63df-kube-api-access-72nr9\") pod \"multus-additional-cni-plugins-rvgbl\" (UID: \"49b6af32-4870-47da-98f8-3fb48b1a63df\") " pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.451954 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.451913 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlws8\" (UniqueName: \"kubernetes.io/projected/8202348f-3147-4b1e-bbd1-c476ac7260fb-kube-api-access-nlws8\") pod \"iptables-alerter-wpjnm\" (UID: \"8202348f-3147-4b1e-bbd1-c476ac7260fb\") " pod="openshift-network-operator/iptables-alerter-wpjnm" Sep 11 14:15:14.453643 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.453536 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb2tn\" (UniqueName: \"kubernetes.io/projected/909ffdc3-b00a-4ae3-b184-5381ab8e7d13-kube-api-access-tb2tn\") pod \"ovnkube-node-8k54l\" (UID: \"909ffdc3-b00a-4ae3-b184-5381ab8e7d13\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.465246 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.465196 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" event={"ID":"d29ff600cf421cf8453088acb7f44e53","Type":"ContainerStarted","Data":"0d5395547c7d16c255cf7a38f30fd88934f0dc4c85ebeb9a5793aa1a23b4bd55"} Sep 11 14:15:14.466837 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.466811 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal" event={"ID":"ffbb18c601b83060fb6937c4ad273ac5","Type":"ContainerStarted","Data":"c1e347ea5724e1e03e84068c0ca07f3ffdf1fac30e9946e031ed68ac66a42500"} Sep 11 14:15:14.527123 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.527090 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vszlk" Sep 11 14:15:14.537373 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.537352 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rvgbl" Sep 11 14:15:14.549208 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.549181 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wpjnm" Sep 11 14:15:14.557223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.557198 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:14.567374 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.567355 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" Sep 11 14:15:14.576249 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.576230 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7plbw" Sep 11 14:15:14.585181 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.585161 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b78k9" Sep 11 14:15:14.593209 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.593190 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2fj2j" Sep 11 14:15:14.600985 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.600965 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6pfsh" Sep 11 14:15:14.934170 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.934074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:14.934170 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:14.934144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:14.934386 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.934227 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:14.934386 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.934245 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:14.934386 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.934302 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs podName:df1160f9-2ec0-4907-9ca2-4b269875d943 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:15.934280671 +0000 UTC m=+4.019737027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs") pod "network-metrics-daemon-pp2sq" (UID: "df1160f9-2ec0-4907-9ca2-4b269875d943") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:14.934386 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:14.934324 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret podName:b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:15.93431281 +0000 UTC m=+4.019769150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret") pod "global-pull-secret-syncer-crnx9" (UID: "b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:15.035158 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:15.035122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7cj\" (UniqueName: \"kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj\") pod \"network-check-target-75q2d\" (UID: \"c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f\") " pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:15.035356 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:15.035278 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:15.035356 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:15.035302 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:15.035356 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:15.035315 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wh7cj for pod openshift-network-diagnostics/network-check-target-75q2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:15.035536 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:15.035382 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj podName:c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:16.035363131 +0000 UTC m=+4.120819488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wh7cj" (UniqueName: "kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj") pod "network-check-target-75q2d" (UID: "c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:15.281379 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:15.281263 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:10:14 +0000 UTC" deadline="2028-06-22 06:47:14.534280946 +0000 UTC" Sep 11 14:15:15.281379 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:15.281320 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15592h31m59.252963263s" Sep 11 14:15:15.457682 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:15.457646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:15.457874 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:15.457802 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:15.586309 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:15.586236 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8202348f_3147_4b1e_bbd1_c476ac7260fb.slice/crio-5480e989b241f617c216492451e27011681715ef123c04dd8f1dbf1e43a0e7ad WatchSource:0}: Error finding container 5480e989b241f617c216492451e27011681715ef123c04dd8f1dbf1e43a0e7ad: Status 404 returned error can't find the container with id 5480e989b241f617c216492451e27011681715ef123c04dd8f1dbf1e43a0e7ad Sep 11 14:15:15.588358 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:15.588331 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda062adf8_6207_40d7_8eb0_302fede4913a.slice/crio-d1f5ac151507dd807dfa9f8a760162d9de675aef3efc1cf1ff3663279993fe05 WatchSource:0}: Error finding container d1f5ac151507dd807dfa9f8a760162d9de675aef3efc1cf1ff3663279993fe05: Status 404 returned error can't find the container with id d1f5ac151507dd807dfa9f8a760162d9de675aef3efc1cf1ff3663279993fe05 Sep 11 14:15:15.591467 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:15.591439 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909ffdc3_b00a_4ae3_b184_5381ab8e7d13.slice/crio-62b8c7c2e0a841d44593b735a5d21ec09480ae7cfeeb55f84b503059638f403e WatchSource:0}: Error finding container 62b8c7c2e0a841d44593b735a5d21ec09480ae7cfeeb55f84b503059638f403e: Status 404 returned error can't find the container with id 62b8c7c2e0a841d44593b735a5d21ec09480ae7cfeeb55f84b503059638f403e Sep 11 14:15:15.592390 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:15.592364 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1994440_b56e_4b0c_b414_f68944e12f56.slice/crio-65469811ab0a0904630c4a93cd474be77e918527eec815ad16c9219585bd790f WatchSource:0}: Error finding container 65469811ab0a0904630c4a93cd474be77e918527eec815ad16c9219585bd790f: Status 404 returned error can't find the container with id 65469811ab0a0904630c4a93cd474be77e918527eec815ad16c9219585bd790f Sep 11 14:15:15.593540 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:15.593175 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b6af32_4870_47da_98f8_3fb48b1a63df.slice/crio-7792abf68955fe54c7bb9a967b5e50a46b885e0acf846e441d1a6fe5f66f62b2 WatchSource:0}: Error finding container 7792abf68955fe54c7bb9a967b5e50a46b885e0acf846e441d1a6fe5f66f62b2: Status 404 returned error can't find the container with id 7792abf68955fe54c7bb9a967b5e50a46b885e0acf846e441d1a6fe5f66f62b2 Sep 11 14:15:15.596651 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:15.596444 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac8086f_f464_4481_88bb_ada601e7351d.slice/crio-0f3858450799b12501bbf3fd9accecfd086d5b14ca7200a855c3f6a0fc17f9ef WatchSource:0}: Error finding container 0f3858450799b12501bbf3fd9accecfd086d5b14ca7200a855c3f6a0fc17f9ef: Status 404 returned error can't find the container with id 0f3858450799b12501bbf3fd9accecfd086d5b14ca7200a855c3f6a0fc17f9ef Sep 11 14:15:15.598677 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:15.598655 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4dec9ea_724d_422e_8783_4795fe390f80.slice/crio-defb218fc10baa2c2285573c6c319de7cfc5ce783196fd19414cc133ea19b545 WatchSource:0}: Error finding container defb218fc10baa2c2285573c6c319de7cfc5ce783196fd19414cc133ea19b545: Status 404 returned error can't find the container with id defb218fc10baa2c2285573c6c319de7cfc5ce783196fd19414cc133ea19b545 Sep 11 14:15:15.599648 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:15.599622 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230f41e9_12d5_4191_9eef_f5af5c5090f8.slice/crio-742187174e9f513475a8aaecac4cf425aba011a449280ca5b16a8ba792a2c5e7 WatchSource:0}: Error finding container 742187174e9f513475a8aaecac4cf425aba011a449280ca5b16a8ba792a2c5e7: Status 404 returned error can't find the container with id 742187174e9f513475a8aaecac4cf425aba011a449280ca5b16a8ba792a2c5e7 Sep 11 14:15:15.941926 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:15.941719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:15.942066 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:15.941939 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:15.942066 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:15.941882 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:15.942066 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:15.942035 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:15.942066 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:15.942060 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs podName:df1160f9-2ec0-4907-9ca2-4b269875d943 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:17.942042545 +0000 UTC m=+6.027498907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs") pod "network-metrics-daemon-pp2sq" (UID: "df1160f9-2ec0-4907-9ca2-4b269875d943") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:15.942206 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:15.942078 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret podName:b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:17.942066693 +0000 UTC m=+6.027523035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret") pod "global-pull-secret-syncer-crnx9" (UID: "b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:16.042847 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.042809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7cj\" (UniqueName: \"kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj\") pod \"network-check-target-75q2d\" (UID: \"c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f\") " pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:16.043044 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:16.043014 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:16.043044 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:16.043045 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:16.043299 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:16.043056 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wh7cj for pod openshift-network-diagnostics/network-check-target-75q2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:16.043299 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:16.043111 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj podName:c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:18.043092814 +0000 UTC m=+6.128549154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wh7cj" (UniqueName: "kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj") pod "network-check-target-75q2d" (UID: "c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:16.110955 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.110734 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:15:16.281758 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.281709 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:10:14 +0000 UTC" deadline="2028-04-18 04:34:29.467142348 +0000 UTC" Sep 11 14:15:16.281758 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.281755 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14030h19m13.185391494s" Sep 11 14:15:16.461358 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.460591 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:16.461358 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:16.460709 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:16.461358 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.461094 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:16.461358 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:16.461211 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:16.475715 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.475653 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vszlk" event={"ID":"c1994440-b56e-4b0c-b414-f68944e12f56","Type":"ContainerStarted","Data":"65469811ab0a0904630c4a93cd474be77e918527eec815ad16c9219585bd790f"} Sep 11 14:15:16.479010 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.478879 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" event={"ID":"909ffdc3-b00a-4ae3-b184-5381ab8e7d13","Type":"ContainerStarted","Data":"62b8c7c2e0a841d44593b735a5d21ec09480ae7cfeeb55f84b503059638f403e"} Sep 11 14:15:16.487266 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.487202 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wpjnm" event={"ID":"8202348f-3147-4b1e-bbd1-c476ac7260fb","Type":"ContainerStarted","Data":"5480e989b241f617c216492451e27011681715ef123c04dd8f1dbf1e43a0e7ad"} Sep 11 14:15:16.494854 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.494822 2578 generic.go:356] "Generic (PLEG): container finished" podID="d29ff600cf421cf8453088acb7f44e53" containerID="0e212ebc0be4a20dff68c1b33fdf564363bcc9436afe9b166c0c9380c3e97f3a" exitCode=0 Sep 11 14:15:16.495254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.495081 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" event={"ID":"d29ff600cf421cf8453088acb7f44e53","Type":"ContainerDied","Data":"0e212ebc0be4a20dff68c1b33fdf564363bcc9436afe9b166c0c9380c3e97f3a"} Sep 11 14:15:16.500814 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.500788 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b78k9" event={"ID":"230f41e9-12d5-4191-9eef-f5af5c5090f8","Type":"ContainerStarted","Data":"742187174e9f513475a8aaecac4cf425aba011a449280ca5b16a8ba792a2c5e7"} Sep 11 14:15:16.506723 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.506654 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" event={"ID":"5647f838-aa18-4f4e-9d6d-522d71950a7c","Type":"ContainerStarted","Data":"74728ee4f970e0d4f10f82a12c2ec80b565d6c426e6825bb49ac447c33cb1e07"} Sep 11 14:15:16.511254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.511187 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2fj2j" event={"ID":"a062adf8-6207-40d7-8eb0-302fede4913a","Type":"ContainerStarted","Data":"d1f5ac151507dd807dfa9f8a760162d9de675aef3efc1cf1ff3663279993fe05"} Sep 11 14:15:16.516454 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.516111 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal" event={"ID":"ffbb18c601b83060fb6937c4ad273ac5","Type":"ContainerStarted","Data":"0dfae67e6348f6b883ebe98b2267051d8de22e17016a73ed6611a603fe4f0629"} Sep 11 14:15:16.520193 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.520131 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7plbw" event={"ID":"d4dec9ea-724d-422e-8783-4795fe390f80","Type":"ContainerStarted","Data":"defb218fc10baa2c2285573c6c319de7cfc5ce783196fd19414cc133ea19b545"} Sep 11 14:15:16.524814 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.524748 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6pfsh" event={"ID":"dac8086f-f464-4481-88bb-ada601e7351d","Type":"ContainerStarted","Data":"0f3858450799b12501bbf3fd9accecfd086d5b14ca7200a855c3f6a0fc17f9ef"} Sep 11 14:15:16.527741 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:16.527647 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvgbl" event={"ID":"49b6af32-4870-47da-98f8-3fb48b1a63df","Type":"ContainerStarted","Data":"7792abf68955fe54c7bb9a967b5e50a46b885e0acf846e441d1a6fe5f66f62b2"} Sep 11 14:15:17.457814 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:17.457782 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:17.458250 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:17.457915 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:17.552708 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:17.552632 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" event={"ID":"d29ff600cf421cf8453088acb7f44e53","Type":"ContainerStarted","Data":"a72d4c3e0d975a16adca38596ed77bc76c7475e27f5317873cb4f4dc7b34df1c"} Sep 11 14:15:17.575752 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:17.574954 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-57.ec2.internal" podStartSLOduration=4.574935253 podStartE2EDuration="4.574935253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:16.536948766 +0000 UTC m=+4.622405128" watchObservedRunningTime="2026-09-11 14:15:17.574935253 +0000 UTC m=+5.660391616" Sep 11 14:15:17.575752 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:17.575500 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-57.ec2.internal" podStartSLOduration=4.575487665 podStartE2EDuration="4.575487665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:17.57451528 +0000 UTC m=+5.659971652" watchObservedRunningTime="2026-09-11 14:15:17.575487665 +0000 UTC m=+5.660944026" Sep 11 14:15:17.962223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:17.962007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:17.962223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:17.962056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:17.962223 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:17.962183 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:17.962510 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:17.962259 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs podName:df1160f9-2ec0-4907-9ca2-4b269875d943 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:21.962239464 +0000 UTC m=+10.047695822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs") pod "network-metrics-daemon-pp2sq" (UID: "df1160f9-2ec0-4907-9ca2-4b269875d943") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:17.962577 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:17.962194 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:17.962631 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:17.962618 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret podName:b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:21.962601995 +0000 UTC m=+10.048058339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret") pod "global-pull-secret-syncer-crnx9" (UID: "b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:18.063246 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:18.062597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7cj\" (UniqueName: \"kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj\") pod \"network-check-target-75q2d\" (UID: \"c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f\") " pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:18.063246 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:18.062806 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:18.063246 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:18.062828 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:18.063246 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:18.062840 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wh7cj for pod openshift-network-diagnostics/network-check-target-75q2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:18.063246 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:18.062900 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj podName:c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:22.062880586 +0000 UTC m=+10.148336925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wh7cj" (UniqueName: "kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj") pod "network-check-target-75q2d" (UID: "c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:18.460355 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:18.459832 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:18.460355 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:18.459965 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:18.461119 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:18.460942 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:18.461119 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:18.461055 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:19.457962 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:19.457927 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:19.458196 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:19.458062 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:20.460845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:20.460801 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:20.461300 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:20.460929 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:20.461300 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:20.460990 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:20.461300 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:20.461079 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:21.457949 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:21.457907 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:21.458138 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:21.458035 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:21.995447 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:21.994963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:21.995447 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:21.995017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:21.995447 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:21.995190 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:21.995447 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:21.995251 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret podName:b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:29.995233677 +0000 UTC m=+18.080690028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret") pod "global-pull-secret-syncer-crnx9" (UID: "b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:21.995447 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:21.995280 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:21.995447 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:21.995319 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs podName:df1160f9-2ec0-4907-9ca2-4b269875d943 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:29.99530787 +0000 UTC m=+18.080764223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs") pod "network-metrics-daemon-pp2sq" (UID: "df1160f9-2ec0-4907-9ca2-4b269875d943") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:22.096009 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:22.095407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7cj\" (UniqueName: \"kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj\") pod \"network-check-target-75q2d\" (UID: \"c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f\") " pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:22.096009 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:22.095580 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:22.096009 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:22.095601 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:22.096009 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:22.095614 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wh7cj for pod openshift-network-diagnostics/network-check-target-75q2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:22.096009 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:22.095670 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj podName:c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:30.095652034 +0000 UTC m=+18.181108377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wh7cj" (UniqueName: "kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj") pod "network-check-target-75q2d" (UID: "c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:22.458940 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:22.458806 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:22.458940 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:22.458806 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:22.458940 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:22.458915 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:22.459215 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:22.459011 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:23.457839 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:23.457800 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:23.458217 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:23.457952 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:24.458127 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:24.458054 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:24.459218 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:24.458703 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:24.459218 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:24.458797 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:24.459218 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:24.458890 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:25.457788 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:25.457758 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:25.457936 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:25.457865 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:26.458195 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:26.458159 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:26.458654 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:26.458302 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:26.458654 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:26.458318 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:26.458654 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:26.458408 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:27.457889 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:27.457850 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:27.458103 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:27.457985 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:28.458559 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:28.458522 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:28.459035 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:28.458658 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:28.459035 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:28.458719 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:28.459035 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:28.458844 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:29.458556 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:29.458520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:29.458822 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:29.458658 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:30.055962 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:30.055919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:30.056142 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:30.056019 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:30.056142 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:30.056084 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:30.056142 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:30.056129 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:30.056284 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:30.056164 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret podName:b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:46.056145224 +0000 UTC m=+34.141601566 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret") pod "global-pull-secret-syncer-crnx9" (UID: "b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:30.056284 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:30.056184 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs podName:df1160f9-2ec0-4907-9ca2-4b269875d943 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:46.056174468 +0000 UTC m=+34.141630810 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs") pod "network-metrics-daemon-pp2sq" (UID: "df1160f9-2ec0-4907-9ca2-4b269875d943") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:30.156471 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:30.156435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7cj\" (UniqueName: \"kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj\") pod \"network-check-target-75q2d\" (UID: \"c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f\") " pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:30.156641 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:30.156577 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:30.156641 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:30.156596 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:30.156641 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:30.156608 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wh7cj for pod openshift-network-diagnostics/network-check-target-75q2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:30.156842 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:30.156665 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj podName:c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:46.15664903 +0000 UTC m=+34.242105387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wh7cj" (UniqueName: "kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj") pod "network-check-target-75q2d" (UID: "c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:30.458662 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:30.458617 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:30.458839 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:30.458617 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:30.458839 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:30.458761 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:30.459297 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:30.458874 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:31.458404 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:31.458368 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:31.458589 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:31.458490 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:32.458909 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:32.458876 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:32.459343 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:32.458988 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:32.459343 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:32.459024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:32.459343 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:32.459165 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:33.458667 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:33.458500 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:33.458667 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:33.458625 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:33.586705 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:33.586479 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b78k9" event={"ID":"230f41e9-12d5-4191-9eef-f5af5c5090f8","Type":"ContainerStarted","Data":"1bd97c67cd9e1879dc015ceabb20714c8c1e16a16524733c42ee6db4b5289352"} Sep 11 14:15:33.588483 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:33.588460 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" event={"ID":"5647f838-aa18-4f4e-9d6d-522d71950a7c","Type":"ContainerStarted","Data":"3c1e2689237a7f68f5f5c95b7efd42d94b615a49dca3b8d42a673afb3795d997"} Sep 11 14:15:33.589995 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:33.589972 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2fj2j" event={"ID":"a062adf8-6207-40d7-8eb0-302fede4913a","Type":"ContainerStarted","Data":"f05bd86fd891a64cf6aea21d0dafdcc0f0288221b05b2b646f20c8543792c6ea"} Sep 11 14:15:33.591996 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:33.591892 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7plbw" event={"ID":"d4dec9ea-724d-422e-8783-4795fe390f80","Type":"ContainerStarted","Data":"6708860cd1d01de974795baff64e723129f570113d6bdbef210dde87bb993fe5"} Sep 11 14:15:33.593506 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:33.593487 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvgbl" event={"ID":"49b6af32-4870-47da-98f8-3fb48b1a63df","Type":"ContainerStarted","Data":"3020e4a9590f6f80658f4b496ad4bd873692fda666e3da4b3857d0466e758a86"} Sep 11 14:15:33.595345 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:33.595319 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vszlk" event={"ID":"c1994440-b56e-4b0c-b414-f68944e12f56","Type":"ContainerStarted","Data":"8837ae672060bb064d34f17f76456411036881893277aadd75c981db7e11a09e"} Sep 11 14:15:33.597578 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:33.597555 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" event={"ID":"909ffdc3-b00a-4ae3-b184-5381ab8e7d13","Type":"ContainerStarted","Data":"d8a79e00babd0466fbee16d1813fff1b0d49b9f766d39bda27f4cf0cd2c2be7b"} Sep 11 14:15:33.628237 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:33.628196 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b78k9" podStartSLOduration=3.9231587279999998 podStartE2EDuration="21.628181322s" totalImagesPullingTime="17.705022594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:33.610477375 +0000 UTC m=+21.695933735" watchObservedRunningTime="2026-09-11 14:15:33.628181322 +0000 UTC m=+21.713637673" Sep 11 14:15:33.628521 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:33.628452 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7plbw" podStartSLOduration=3.923978727 podStartE2EDuration="21.628447438s" totalImagesPullingTime="17.704468711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:33.627963977 +0000 UTC m=+21.713420338" watchObservedRunningTime="2026-09-11 14:15:33.628447438 +0000 UTC m=+21.713903797" Sep 11 14:15:33.677442 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:33.677253 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vszlk" podStartSLOduration=11.380754571 podStartE2EDuration="21.677237669s" totalImagesPullingTime="10.296483098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:33.648619849 +0000 UTC m=+21.734076212" watchObservedRunningTime="2026-09-11 14:15:33.677237669 +0000 UTC m=+21.762694081" Sep 11 14:15:34.457874 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.457845 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:34.458020 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.457847 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:34.458020 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:34.457947 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:34.458095 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:34.458046 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:34.602989 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.602959 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6pfsh" event={"ID":"dac8086f-f464-4481-88bb-ada601e7351d","Type":"ContainerStarted","Data":"dc883207688aea636ce22ba5c0d2002d69924760b6dfef312aa4efee037bd982"} Sep 11 14:15:34.604493 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.604465 2578 generic.go:356] "Generic (PLEG): container finished" podID="49b6af32-4870-47da-98f8-3fb48b1a63df" containerID="3020e4a9590f6f80658f4b496ad4bd873692fda666e3da4b3857d0466e758a86" exitCode=0 Sep 11 14:15:34.604611 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.604552 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvgbl" event={"ID":"49b6af32-4870-47da-98f8-3fb48b1a63df","Type":"ContainerDied","Data":"3020e4a9590f6f80658f4b496ad4bd873692fda666e3da4b3857d0466e758a86"} Sep 11 14:15:34.607394 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.607374 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:15:34.607723 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.607705 2578 generic.go:356] "Generic (PLEG): container finished" podID="909ffdc3-b00a-4ae3-b184-5381ab8e7d13" containerID="412863bd41bc87f33b082292479f843a6fd99b01bf514459adf927a19a6b6045" exitCode=1 Sep 11 14:15:34.607849 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.607754 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" event={"ID":"909ffdc3-b00a-4ae3-b184-5381ab8e7d13","Type":"ContainerStarted","Data":"18082146392b4ecd26a8818d6c93b5b4a764a2f10c4275163390783886fd893d"} Sep 11 14:15:34.607849 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.607806 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" event={"ID":"909ffdc3-b00a-4ae3-b184-5381ab8e7d13","Type":"ContainerStarted","Data":"3eea75b554d31b670e37165b93ea1c4cd996b74a70fb6637944cb821011e8340"} Sep 11 14:15:34.607952 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.607847 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" event={"ID":"909ffdc3-b00a-4ae3-b184-5381ab8e7d13","Type":"ContainerStarted","Data":"ff7c236ba282c8e8ee2538e09282d6b2c7c339b4aa94832cb537aad9cc29d31f"} Sep 11 14:15:34.607952 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.607893 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" event={"ID":"909ffdc3-b00a-4ae3-b184-5381ab8e7d13","Type":"ContainerStarted","Data":"ce5c072aa05584498b462e204b5a1a5eb5bf3ad71908eb84826836103288aa23"} Sep 11 14:15:34.607952 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.607931 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" event={"ID":"909ffdc3-b00a-4ae3-b184-5381ab8e7d13","Type":"ContainerDied","Data":"412863bd41bc87f33b082292479f843a6fd99b01bf514459adf927a19a6b6045"} Sep 11 14:15:34.609077 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.609059 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wpjnm" event={"ID":"8202348f-3147-4b1e-bbd1-c476ac7260fb","Type":"ContainerStarted","Data":"8e111528f784dd41a1f9562caa1b58f5958c3d4881dddc247d5bc062d5eddd16"} Sep 11 14:15:34.623996 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.623955 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2fj2j" podStartSLOduration=4.906886028 podStartE2EDuration="22.62394329s" totalImagesPullingTime="17.717057262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:33.695623725 +0000 UTC m=+21.781080086" watchObservedRunningTime="2026-09-11 14:15:34.62394329 +0000 UTC m=+22.709399651" Sep 11 14:15:34.624141 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.624039 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6pfsh" podStartSLOduration=4.667369427 podStartE2EDuration="22.624032091s" totalImagesPullingTime="17.956662664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:34.623809469 +0000 UTC m=+22.709265830" watchObservedRunningTime="2026-09-11 14:15:34.624032091 +0000 UTC m=+22.709488452" Sep 11 14:15:34.673033 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:34.672990 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wpjnm" podStartSLOduration=4.955519481 podStartE2EDuration="22.672975402s" totalImagesPullingTime="17.717455921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:34.672782669 +0000 UTC m=+22.758239028" watchObservedRunningTime="2026-09-11 14:15:34.672975402 +0000 UTC m=+22.758431762" Sep 11 14:15:35.458405 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:35.458383 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:35.458532 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:35.458504 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:35.588896 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:35.588816 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 14:15:35.613589 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:35.613558 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" event={"ID":"5647f838-aa18-4f4e-9d6d-522d71950a7c","Type":"ContainerStarted","Data":"abca8d50b6cf73945a81fb5592435f6466dd141797d260577f4805c3d5e5ebf3"} Sep 11 14:15:36.278231 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:36.278062 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T14:15:35.588836618Z","UUID":"2d5124d6-315d-4efb-94cc-5b532260870c","Handler":null,"Name":"","Endpoint":""} Sep 11 14:15:36.279892 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:36.279868 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 14:15:36.280016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:36.279905 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 14:15:36.458467 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:36.458437 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:36.458601 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:36.458580 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:36.458989 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:36.458963 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:36.459096 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:36.459056 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:36.619390 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:36.619364 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:15:36.619878 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:36.619856 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" event={"ID":"909ffdc3-b00a-4ae3-b184-5381ab8e7d13","Type":"ContainerStarted","Data":"540a033a06592ff804bc48c9a1e5c90f66ed39e32fa48feef1c4ebb022424b67"} Sep 11 14:15:36.622095 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:36.622072 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" event={"ID":"5647f838-aa18-4f4e-9d6d-522d71950a7c","Type":"ContainerStarted","Data":"a9fe8e5e2af2846f514f3ea24b511297dd3a502617da7117ed273d7d0fa1552b"} Sep 11 14:15:36.645262 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:36.645211 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vnwqx" podStartSLOduration=4.215978637 podStartE2EDuration="24.645192892s" totalImagesPullingTime="20.429214255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:15:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:36.644620391 +0000 UTC m=+24.730076750" watchObservedRunningTime="2026-09-11 14:15:36.645192892 +0000 UTC m=+24.730649255" Sep 11 14:15:37.458087 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:37.458050 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:37.458302 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:37.458179 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:38.288382 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:38.288342 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vszlk" Sep 11 14:15:38.288931 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:38.288883 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vszlk" Sep 11 14:15:38.457807 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:38.457775 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:38.457985 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:38.457775 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:38.457985 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:38.457966 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:38.458187 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:38.458162 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:38.626595 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:38.626562 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vszlk" Sep 11 14:15:38.627027 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:38.627009 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vszlk" Sep 11 14:15:39.458068 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:39.457857 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:39.458600 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:39.458099 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:39.630425 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:39.630393 2578 generic.go:356] "Generic (PLEG): container finished" podID="49b6af32-4870-47da-98f8-3fb48b1a63df" containerID="13eff88739ff1123f45a216bc271b4a5f66d7b4e341b0499bc52048a349ec33b" exitCode=0 Sep 11 14:15:39.630581 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:39.630474 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvgbl" event={"ID":"49b6af32-4870-47da-98f8-3fb48b1a63df","Type":"ContainerDied","Data":"13eff88739ff1123f45a216bc271b4a5f66d7b4e341b0499bc52048a349ec33b"} Sep 11 14:15:39.633888 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:39.633871 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:15:39.634277 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:39.634255 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" event={"ID":"909ffdc3-b00a-4ae3-b184-5381ab8e7d13","Type":"ContainerStarted","Data":"e10596d38c7a7f46056685ae7a5bca86cf015ab2d11a08e89e92a0ad012f23df"} Sep 11 14:15:39.634641 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:39.634625 2578 scope.go:117] "RemoveContainer" containerID="412863bd41bc87f33b082292479f843a6fd99b01bf514459adf927a19a6b6045" Sep 11 14:15:40.457542 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:40.457511 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:40.457819 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:40.457555 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:40.457819 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:40.457629 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:40.457819 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:40.457745 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:40.640332 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:40.640307 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:15:40.640961 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:40.640677 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" event={"ID":"909ffdc3-b00a-4ae3-b184-5381ab8e7d13","Type":"ContainerStarted","Data":"23ba166d89b96b95c0203af8b7aa27b2ddc387952223b9f567b2e25ab7b03e40"} Sep 11 14:15:40.640961 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:40.640903 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:40.640961 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:40.640941 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:40.641157 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:40.640978 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:40.656081 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:40.656047 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:40.661754 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:40.661717 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:15:40.678232 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:40.678195 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" podStartSLOduration=10.924537458 podStartE2EDuration="28.678181654s" totalImagesPullingTime="17.753644196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:40.677805303 +0000 UTC m=+28.763261664" watchObservedRunningTime="2026-09-11 14:15:40.678181654 +0000 UTC m=+28.763638037" Sep 11 14:15:41.214886 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:41.214696 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-crnx9"] Sep 11 14:15:41.215037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:41.214987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:41.215103 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:41.215082 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:41.217895 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:41.217872 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pp2sq"] Sep 11 14:15:41.218037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:41.218019 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:41.218155 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:41.218137 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:41.218458 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:41.218436 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-75q2d"] Sep 11 14:15:41.218548 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:41.218522 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:41.218607 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:41.218588 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:41.645266 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:41.645232 2578 generic.go:356] "Generic (PLEG): container finished" podID="49b6af32-4870-47da-98f8-3fb48b1a63df" containerID="0e2de22bc6a3f86f9930e574bdfea601f30c39e57f0a510ad5ea92413da20c6c" exitCode=0 Sep 11 14:15:41.645862 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:41.645316 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvgbl" event={"ID":"49b6af32-4870-47da-98f8-3fb48b1a63df","Type":"ContainerDied","Data":"0e2de22bc6a3f86f9930e574bdfea601f30c39e57f0a510ad5ea92413da20c6c"} Sep 11 14:15:42.458760 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:42.458689 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:42.458877 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:42.458791 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:42.458877 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:42.458860 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:42.458980 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:42.458954 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:42.649526 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:42.649494 2578 generic.go:356] "Generic (PLEG): container finished" podID="49b6af32-4870-47da-98f8-3fb48b1a63df" containerID="5fb65a35efba8ada5445581037ef07f8ccfa3211c5b0d1fe303417555ccef0f4" exitCode=0 Sep 11 14:15:42.650209 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:42.649547 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvgbl" event={"ID":"49b6af32-4870-47da-98f8-3fb48b1a63df","Type":"ContainerDied","Data":"5fb65a35efba8ada5445581037ef07f8ccfa3211c5b0d1fe303417555ccef0f4"} Sep 11 14:15:43.458384 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:43.458348 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:43.458549 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:43.458446 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:44.457655 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:44.457622 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:44.458239 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:44.457766 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75q2d" podUID="c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f" Sep 11 14:15:44.458239 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:44.457854 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:44.458239 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:44.457972 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pp2sq" podUID="df1160f9-2ec0-4907-9ca2-4b269875d943" Sep 11 14:15:45.458456 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:45.458385 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:45.458895 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:45.458545 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-crnx9" podUID="b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8" Sep 11 14:15:46.068413 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.068375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:46.068615 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.068434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:46.068615 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:46.068551 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:46.068615 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:46.068598 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:46.068794 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:46.068618 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs podName:df1160f9-2ec0-4907-9ca2-4b269875d943 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:18.068599524 +0000 UTC m=+66.154055889 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs") pod "network-metrics-daemon-pp2sq" (UID: "df1160f9-2ec0-4907-9ca2-4b269875d943") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:46.068794 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:46.068657 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret podName:b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:18.068639712 +0000 UTC m=+66.154096050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret") pod "global-pull-secret-syncer-crnx9" (UID: "b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:46.169196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.169169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7cj\" (UniqueName: \"kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj\") pod \"network-check-target-75q2d\" (UID: \"c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f\") " pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:46.169329 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:46.169315 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:46.169372 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:46.169339 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:46.169372 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:46.169351 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wh7cj for pod openshift-network-diagnostics/network-check-target-75q2d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:46.169446 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:46.169401 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj podName:c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:18.1693879 +0000 UTC m=+66.254844238 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wh7cj" (UniqueName: "kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj") pod "network-check-target-75q2d" (UID: "c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:46.230888 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.230818 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-57.ec2.internal" event="NodeReady" Sep 11 14:15:46.231035 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.230964 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 14:15:46.291412 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.291383 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qdqt7"] Sep 11 14:15:46.312770 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.312717 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r5jzm"] Sep 11 14:15:46.312907 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.312832 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:46.316487 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.316221 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 14:15:46.316487 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.316265 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c4f62\"" Sep 11 14:15:46.316670 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.316632 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 14:15:46.334793 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.334768 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qdqt7"] Sep 11 14:15:46.334916 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.334813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r5jzm" Sep 11 14:15:46.334916 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.334822 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r5jzm"] Sep 11 14:15:46.338556 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.338281 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xs77v\"" Sep 11 14:15:46.338556 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.338283 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 14:15:46.338556 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.338330 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 14:15:46.338556 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.338396 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 14:15:46.370026 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.370003 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ae826c-e270-4612-9d92-9982ce13b392-tmp-dir\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:46.370148 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.370054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ptc4\" (UniqueName: \"kubernetes.io/projected/e9ae826c-e270-4612-9d92-9982ce13b392-kube-api-access-8ptc4\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:46.370148 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.370082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ae826c-e270-4612-9d92-9982ce13b392-config-volume\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:46.370227 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.370159 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:46.458366 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.458336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:15:46.458553 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.458531 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:15:46.461514 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.461483 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 14:15:46.461770 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.461697 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-96vl6\"" Sep 11 14:15:46.461770 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.461748 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 14:15:46.461770 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.461755 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-brmn7\"" Sep 11 14:15:46.461971 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.461697 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 14:15:46.471094 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.471076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ae826c-e270-4612-9d92-9982ce13b392-tmp-dir\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:46.471205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.471123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ptc4\" (UniqueName: \"kubernetes.io/projected/e9ae826c-e270-4612-9d92-9982ce13b392-kube-api-access-8ptc4\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:46.471205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.471152 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ae826c-e270-4612-9d92-9982ce13b392-config-volume\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:46.471205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.471195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a30eb9bb-451d-4399-8180-10739866c4a4-cert\") pod \"ingress-canary-r5jzm\" (UID: \"a30eb9bb-451d-4399-8180-10739866c4a4\") " pod="openshift-ingress-canary/ingress-canary-r5jzm" Sep 11 14:15:46.471371 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.471223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vht4f\" (UniqueName: \"kubernetes.io/projected/a30eb9bb-451d-4399-8180-10739866c4a4-kube-api-access-vht4f\") pod \"ingress-canary-r5jzm\" (UID: \"a30eb9bb-451d-4399-8180-10739866c4a4\") " pod="openshift-ingress-canary/ingress-canary-r5jzm" Sep 11 14:15:46.471371 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.471251 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:46.471371 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:46.471345 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:46.471515 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:46.471383 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls podName:e9ae826c-e270-4612-9d92-9982ce13b392 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:46.971371589 +0000 UTC m=+35.056827940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls") pod "dns-default-qdqt7" (UID: "e9ae826c-e270-4612-9d92-9982ce13b392") : secret "dns-default-metrics-tls" not found Sep 11 14:15:46.471515 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.471437 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ae826c-e270-4612-9d92-9982ce13b392-tmp-dir\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:46.471794 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.471775 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ae826c-e270-4612-9d92-9982ce13b392-config-volume\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:46.485290 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.485242 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ptc4\" (UniqueName: \"kubernetes.io/projected/e9ae826c-e270-4612-9d92-9982ce13b392-kube-api-access-8ptc4\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:46.572347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.572318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a30eb9bb-451d-4399-8180-10739866c4a4-cert\") pod \"ingress-canary-r5jzm\" (UID: \"a30eb9bb-451d-4399-8180-10739866c4a4\") " pod="openshift-ingress-canary/ingress-canary-r5jzm" Sep 11 14:15:46.572484 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.572351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vht4f\" (UniqueName: \"kubernetes.io/projected/a30eb9bb-451d-4399-8180-10739866c4a4-kube-api-access-vht4f\") pod \"ingress-canary-r5jzm\" (UID: \"a30eb9bb-451d-4399-8180-10739866c4a4\") " pod="openshift-ingress-canary/ingress-canary-r5jzm" Sep 11 14:15:46.575277 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.575254 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a30eb9bb-451d-4399-8180-10739866c4a4-cert\") pod \"ingress-canary-r5jzm\" (UID: \"a30eb9bb-451d-4399-8180-10739866c4a4\") " pod="openshift-ingress-canary/ingress-canary-r5jzm" Sep 11 14:15:46.586466 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.586439 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vht4f\" (UniqueName: \"kubernetes.io/projected/a30eb9bb-451d-4399-8180-10739866c4a4-kube-api-access-vht4f\") pod \"ingress-canary-r5jzm\" (UID: \"a30eb9bb-451d-4399-8180-10739866c4a4\") " pod="openshift-ingress-canary/ingress-canary-r5jzm" Sep 11 14:15:46.646388 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.646360 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r5jzm" Sep 11 14:15:46.974969 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:46.974937 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:46.975162 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:46.975089 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:46.975224 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:46.975163 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls podName:e9ae826c-e270-4612-9d92-9982ce13b392 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:47.975143855 +0000 UTC m=+36.060600212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls") pod "dns-default-qdqt7" (UID: "e9ae826c-e270-4612-9d92-9982ce13b392") : secret "dns-default-metrics-tls" not found Sep 11 14:15:47.457684 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:47.457519 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:15:47.461191 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:47.461173 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 14:15:47.642875 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:47.642317 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r5jzm"] Sep 11 14:15:47.650461 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:15:47.650425 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30eb9bb_451d_4399_8180_10739866c4a4.slice/crio-362944bcb6345dcd885740d24eb093afce74e4d2b75c94d4acfd446ec071876a WatchSource:0}: Error finding container 362944bcb6345dcd885740d24eb093afce74e4d2b75c94d4acfd446ec071876a: Status 404 returned error can't find the container with id 362944bcb6345dcd885740d24eb093afce74e4d2b75c94d4acfd446ec071876a Sep 11 14:15:47.663814 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:47.663785 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r5jzm" event={"ID":"a30eb9bb-451d-4399-8180-10739866c4a4","Type":"ContainerStarted","Data":"362944bcb6345dcd885740d24eb093afce74e4d2b75c94d4acfd446ec071876a"} Sep 11 14:15:47.981502 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:47.981433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:47.981613 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:47.981581 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:47.981650 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:47.981645 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls podName:e9ae826c-e270-4612-9d92-9982ce13b392 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:49.981630567 +0000 UTC m=+38.067086929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls") pod "dns-default-qdqt7" (UID: "e9ae826c-e270-4612-9d92-9982ce13b392") : secret "dns-default-metrics-tls" not found Sep 11 14:15:48.669916 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:48.669885 2578 generic.go:356] "Generic (PLEG): container finished" podID="49b6af32-4870-47da-98f8-3fb48b1a63df" containerID="17d089205a67a7b6e9e73c031a37c9680f749f8f86479514b080a48cf9577fc5" exitCode=0 Sep 11 14:15:48.670300 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:48.669946 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvgbl" event={"ID":"49b6af32-4870-47da-98f8-3fb48b1a63df","Type":"ContainerDied","Data":"17d089205a67a7b6e9e73c031a37c9680f749f8f86479514b080a48cf9577fc5"} Sep 11 14:15:49.676305 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:49.676078 2578 generic.go:356] "Generic (PLEG): container finished" podID="49b6af32-4870-47da-98f8-3fb48b1a63df" containerID="8307dae1912a7c4915dac8fe27bcc10d907711ba0224d7c1ac469d909401732b" exitCode=0 Sep 11 14:15:49.676752 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:49.676163 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvgbl" event={"ID":"49b6af32-4870-47da-98f8-3fb48b1a63df","Type":"ContainerDied","Data":"8307dae1912a7c4915dac8fe27bcc10d907711ba0224d7c1ac469d909401732b"} Sep 11 14:15:49.994363 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:49.994290 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:49.994551 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:49.994467 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:49.994551 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:49.994545 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls podName:e9ae826c-e270-4612-9d92-9982ce13b392 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:53.99452538 +0000 UTC m=+42.079981718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls") pod "dns-default-qdqt7" (UID: "e9ae826c-e270-4612-9d92-9982ce13b392") : secret "dns-default-metrics-tls" not found Sep 11 14:15:50.681204 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:50.681169 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rvgbl" event={"ID":"49b6af32-4870-47da-98f8-3fb48b1a63df","Type":"ContainerStarted","Data":"bc553ee8e247020181e849048cb98391ad8e87b7408fa6c841040a0f6a845d7b"} Sep 11 14:15:50.713443 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:50.713390 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rvgbl" podStartSLOduration=9.835765302 podStartE2EDuration="38.713374913s" totalImagesPullingTime="28.877609611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:15:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:50.712926488 +0000 UTC m=+38.798382851" watchObservedRunningTime="2026-09-11 14:15:50.713374913 +0000 UTC m=+38.798831273" Sep 11 14:15:51.684740 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:51.684704 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r5jzm" event={"ID":"a30eb9bb-451d-4399-8180-10739866c4a4","Type":"ContainerStarted","Data":"f9095dc97dfa7d2c99dce843778002a48947f036afe05af62e36c469423e819a"} Sep 11 14:15:51.714599 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:51.714552 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r5jzm" podStartSLOduration=2.242734454 podStartE2EDuration="5.714534521s" totalImagesPullingTime="3.471800067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:51.713905646 +0000 UTC m=+39.799362020" watchObservedRunningTime="2026-09-11 14:15:51.714534521 +0000 UTC m=+39.799990880" Sep 11 14:15:54.020102 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:15:54.020067 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:15:54.020508 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:54.020226 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:54.020566 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:15:54.020552 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls podName:e9ae826c-e270-4612-9d92-9982ce13b392 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:02.020272294 +0000 UTC m=+50.105728631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls") pod "dns-default-qdqt7" (UID: "e9ae826c-e270-4612-9d92-9982ce13b392") : secret "dns-default-metrics-tls" not found Sep 11 14:16:02.074227 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:02.074184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:16:02.074654 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:02.074360 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:16:02.074654 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:02.074429 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls podName:e9ae826c-e270-4612-9d92-9982ce13b392 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:18.074414122 +0000 UTC m=+66.159870478 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls") pod "dns-default-qdqt7" (UID: "e9ae826c-e270-4612-9d92-9982ce13b392") : secret "dns-default-metrics-tls" not found Sep 11 14:16:09.196814 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.196767 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n"] Sep 11 14:16:09.215787 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.215761 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:09.222494 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.222469 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pxtph\"" Sep 11 14:16:09.222611 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.222497 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 14:16:09.224679 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.224663 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 14:16:09.224806 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.224679 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 11 14:16:09.225490 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.225471 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n"] Sep 11 14:16:09.229509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.229490 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 11 14:16:09.310566 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.310530 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9"] Sep 11 14:16:09.324906 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.324883 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-59b65c65d4-d5j44"] Sep 11 14:16:09.325030 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.325014 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:09.329815 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.329793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:09.329938 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.329830 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wlv7\" (UniqueName: \"kubernetes.io/projected/9008fdf9-a87b-4914-8469-d60013b1f55d-kube-api-access-6wlv7\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:09.329938 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.329874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9008fdf9-a87b-4914-8469-d60013b1f55d-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:09.331062 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.331043 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 11 14:16:09.331709 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.331630 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 11 14:16:09.331709 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.331630 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jdpqx\"" Sep 11 14:16:09.331924 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.331910 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:16:09.338885 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.338866 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59b65c65d4-d5j44"] Sep 11 14:16:09.338988 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.338899 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9"] Sep 11 14:16:09.339049 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.338997 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.341769 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.341749 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 11 14:16:09.342066 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.342046 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 14:16:09.342171 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.342155 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mgqlq\"" Sep 11 14:16:09.342265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.342248 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 14:16:09.342352 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.342337 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 11 14:16:09.342413 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.342337 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 11 14:16:09.342413 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.342343 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 11 14:16:09.430852 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.430824 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-stats-auth\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.430852 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.430854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.431064 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.430877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:09.431064 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.430897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wlv7\" (UniqueName: \"kubernetes.io/projected/9008fdf9-a87b-4914-8469-d60013b1f55d-kube-api-access-6wlv7\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:09.431064 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.430919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-gt6s9\" (UID: \"be9ad885-960c-4e3e-b2c1-ecc75a2010c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:09.431064 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.430949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.431064 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.430990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9008fdf9-a87b-4914-8469-d60013b1f55d-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:09.431064 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:09.431015 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 14:16:09.431064 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.431054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4c58\" (UniqueName: \"kubernetes.io/projected/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-kube-api-access-v4c58\") pod \"cluster-samples-operator-86499c89c4-gt6s9\" (UID: \"be9ad885-960c-4e3e-b2c1-ecc75a2010c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:09.431377 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:09.431086 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls podName:9008fdf9-a87b-4914-8469-d60013b1f55d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:09.931066891 +0000 UTC m=+58.016523229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-47h9n" (UID: "9008fdf9-a87b-4914-8469-d60013b1f55d") : secret "cluster-monitoring-operator-tls" not found Sep 11 14:16:09.431377 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.431160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdwv\" (UniqueName: \"kubernetes.io/projected/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-kube-api-access-hxdwv\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.431377 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.431197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-default-certificate\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.431651 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.431631 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9008fdf9-a87b-4914-8469-d60013b1f55d-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:09.454497 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.454435 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wlv7\" (UniqueName: \"kubernetes.io/projected/9008fdf9-a87b-4914-8469-d60013b1f55d-kube-api-access-6wlv7\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:09.532174 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.532144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.532309 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.532199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4c58\" (UniqueName: \"kubernetes.io/projected/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-kube-api-access-v4c58\") pod \"cluster-samples-operator-86499c89c4-gt6s9\" (UID: \"be9ad885-960c-4e3e-b2c1-ecc75a2010c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:09.532309 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.532228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdwv\" (UniqueName: \"kubernetes.io/projected/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-kube-api-access-hxdwv\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.532401 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:09.532310 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle podName:c53e1dae-b5f7-468f-a5d3-17cb726fcd33 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:10.032293596 +0000 UTC m=+58.117749934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle") pod "router-default-59b65c65d4-d5j44" (UID: "c53e1dae-b5f7-468f-a5d3-17cb726fcd33") : configmap references non-existent config key: service-ca.crt Sep 11 14:16:09.532401 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.532337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-default-certificate\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.532401 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.532379 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-stats-auth\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.532577 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.532403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.532577 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.532435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-gt6s9\" (UID: \"be9ad885-960c-4e3e-b2c1-ecc75a2010c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:09.532577 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:09.532505 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 14:16:09.532577 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:09.532537 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls podName:be9ad885-960c-4e3e-b2c1-ecc75a2010c7 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:10.03252889 +0000 UTC m=+58.117985228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-gt6s9" (UID: "be9ad885-960c-4e3e-b2c1-ecc75a2010c7") : secret "samples-operator-tls" not found Sep 11 14:16:09.532577 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:09.532554 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 14:16:09.532787 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:09.532608 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs podName:c53e1dae-b5f7-468f-a5d3-17cb726fcd33 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:10.032591069 +0000 UTC m=+58.118047412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs") pod "router-default-59b65c65d4-d5j44" (UID: "c53e1dae-b5f7-468f-a5d3-17cb726fcd33") : secret "router-metrics-certs-default" not found Sep 11 14:16:09.534660 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.534644 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-stats-auth\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.534717 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.534701 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-default-certificate\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.546509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.546488 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4c58\" (UniqueName: \"kubernetes.io/projected/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-kube-api-access-v4c58\") pod \"cluster-samples-operator-86499c89c4-gt6s9\" (UID: \"be9ad885-960c-4e3e-b2c1-ecc75a2010c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:09.547354 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.547336 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdwv\" (UniqueName: \"kubernetes.io/projected/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-kube-api-access-hxdwv\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:09.936191 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:09.936149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:09.936400 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:09.936322 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 14:16:09.936464 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:09.936422 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls podName:9008fdf9-a87b-4914-8469-d60013b1f55d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:10.936397356 +0000 UTC m=+59.021853699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-47h9n" (UID: "9008fdf9-a87b-4914-8469-d60013b1f55d") : secret "cluster-monitoring-operator-tls" not found Sep 11 14:16:10.037400 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:10.037360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:10.037596 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:10.037421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-gt6s9\" (UID: \"be9ad885-960c-4e3e-b2c1-ecc75a2010c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:10.037596 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:10.037456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:10.037596 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:10.037518 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 14:16:10.037596 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:10.037568 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 14:16:10.037778 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:10.037609 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs podName:c53e1dae-b5f7-468f-a5d3-17cb726fcd33 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:11.037589802 +0000 UTC m=+59.123046152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs") pod "router-default-59b65c65d4-d5j44" (UID: "c53e1dae-b5f7-468f-a5d3-17cb726fcd33") : secret "router-metrics-certs-default" not found Sep 11 14:16:10.037778 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:10.037627 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls podName:be9ad885-960c-4e3e-b2c1-ecc75a2010c7 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:11.03761949 +0000 UTC m=+59.123075829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-gt6s9" (UID: "be9ad885-960c-4e3e-b2c1-ecc75a2010c7") : secret "samples-operator-tls" not found Sep 11 14:16:10.037778 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:10.037644 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle podName:c53e1dae-b5f7-468f-a5d3-17cb726fcd33 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:11.037636242 +0000 UTC m=+59.123092607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle") pod "router-default-59b65c65d4-d5j44" (UID: "c53e1dae-b5f7-468f-a5d3-17cb726fcd33") : configmap references non-existent config key: service-ca.crt Sep 11 14:16:10.944323 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:10.944288 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:10.944690 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:10.944446 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 14:16:10.944690 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:10.944533 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls podName:9008fdf9-a87b-4914-8469-d60013b1f55d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:12.94451949 +0000 UTC m=+61.029975846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-47h9n" (UID: "9008fdf9-a87b-4914-8469-d60013b1f55d") : secret "cluster-monitoring-operator-tls" not found Sep 11 14:16:11.044935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:11.044897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:11.045084 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:11.045067 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle podName:c53e1dae-b5f7-468f-a5d3-17cb726fcd33 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:13.045050532 +0000 UTC m=+61.130506870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle") pod "router-default-59b65c65d4-d5j44" (UID: "c53e1dae-b5f7-468f-a5d3-17cb726fcd33") : configmap references non-existent config key: service-ca.crt Sep 11 14:16:11.045207 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:11.045095 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:11.045207 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:11.045131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-gt6s9\" (UID: \"be9ad885-960c-4e3e-b2c1-ecc75a2010c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:11.045315 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:11.045216 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 14:16:11.045315 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:11.045228 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 14:16:11.045315 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:11.045253 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls podName:be9ad885-960c-4e3e-b2c1-ecc75a2010c7 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:13.045241358 +0000 UTC m=+61.130697715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-gt6s9" (UID: "be9ad885-960c-4e3e-b2c1-ecc75a2010c7") : secret "samples-operator-tls" not found Sep 11 14:16:11.045315 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:11.045287 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs podName:c53e1dae-b5f7-468f-a5d3-17cb726fcd33 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:13.045270617 +0000 UTC m=+61.130726961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs") pod "router-default-59b65c65d4-d5j44" (UID: "c53e1dae-b5f7-468f-a5d3-17cb726fcd33") : secret "router-metrics-certs-default" not found Sep 11 14:16:12.248356 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.248324 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ddrcm"] Sep 11 14:16:12.250718 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.250702 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.253363 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.253341 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 14:16:12.253556 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.253539 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 14:16:12.254784 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.254763 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 14:16:12.254784 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.254777 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 14:16:12.254936 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.254793 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rj6zn\"" Sep 11 14:16:12.267621 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.267596 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ddrcm"] Sep 11 14:16:12.353428 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.353390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.353604 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.353450 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq8f9\" (UniqueName: \"kubernetes.io/projected/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-api-access-gq8f9\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.353604 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.353569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.353704 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.353677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7958f39-de27-45ff-a3a3-cdc8f7908526-data-volume\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.353816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.353792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7958f39-de27-45ff-a3a3-cdc8f7908526-crio-socket\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.454582 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.454544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.454582 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.454581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq8f9\" (UniqueName: \"kubernetes.io/projected/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-api-access-gq8f9\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.454815 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.454602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.454815 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.454652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7958f39-de27-45ff-a3a3-cdc8f7908526-data-volume\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.454815 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:12.454671 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:12.454815 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.454692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7958f39-de27-45ff-a3a3-cdc8f7908526-crio-socket\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.454815 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:12.454799 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm podName:b7958f39-de27-45ff-a3a3-cdc8f7908526 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:12.954780162 +0000 UTC m=+61.040236500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ddrcm" (UID: "b7958f39-de27-45ff-a3a3-cdc8f7908526") : configmap "kube-rbac-proxy" not found Sep 11 14:16:12.454815 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:12.454800 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:12.455029 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.454830 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7958f39-de27-45ff-a3a3-cdc8f7908526-crio-socket\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.455029 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:12.454846 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls podName:b7958f39-de27-45ff-a3a3-cdc8f7908526 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:12.954837578 +0000 UTC m=+61.040293916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ddrcm" (UID: "b7958f39-de27-45ff-a3a3-cdc8f7908526") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:12.455029 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.455013 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7958f39-de27-45ff-a3a3-cdc8f7908526-data-volume\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.467319 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.467292 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq8f9\" (UniqueName: \"kubernetes.io/projected/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-api-access-gq8f9\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.659505 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.659476 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k54l" Sep 11 14:16:12.959415 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.959323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:12.959558 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.959421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.959558 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:12.959445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:12.959558 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:12.959482 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 14:16:12.959558 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:12.959544 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:12.959558 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:12.959553 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls podName:9008fdf9-a87b-4914-8469-d60013b1f55d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:16.959536692 +0000 UTC m=+65.044993052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-47h9n" (UID: "9008fdf9-a87b-4914-8469-d60013b1f55d") : secret "cluster-monitoring-operator-tls" not found Sep 11 14:16:12.959781 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:12.959589 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:12.959781 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:12.959596 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm podName:b7958f39-de27-45ff-a3a3-cdc8f7908526 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:13.959580702 +0000 UTC m=+62.045037055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ddrcm" (UID: "b7958f39-de27-45ff-a3a3-cdc8f7908526") : configmap "kube-rbac-proxy" not found Sep 11 14:16:12.959781 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:12.959635 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls podName:b7958f39-de27-45ff-a3a3-cdc8f7908526 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:13.959626864 +0000 UTC m=+62.045083207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ddrcm" (UID: "b7958f39-de27-45ff-a3a3-cdc8f7908526") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:13.060600 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:13.060565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:13.060778 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:13.060616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-gt6s9\" (UID: \"be9ad885-960c-4e3e-b2c1-ecc75a2010c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:13.060778 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:13.060711 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 14:16:13.060863 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:13.060781 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 14:16:13.060863 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:13.060787 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls podName:be9ad885-960c-4e3e-b2c1-ecc75a2010c7 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:17.060769734 +0000 UTC m=+65.146226086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-gt6s9" (UID: "be9ad885-960c-4e3e-b2c1-ecc75a2010c7") : secret "samples-operator-tls" not found Sep 11 14:16:13.060863 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:13.060829 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs podName:c53e1dae-b5f7-468f-a5d3-17cb726fcd33 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:17.060818312 +0000 UTC m=+65.146274651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs") pod "router-default-59b65c65d4-d5j44" (UID: "c53e1dae-b5f7-468f-a5d3-17cb726fcd33") : secret "router-metrics-certs-default" not found Sep 11 14:16:13.060863 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:13.060848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:13.061055 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:13.061030 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle podName:c53e1dae-b5f7-468f-a5d3-17cb726fcd33 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:17.061009966 +0000 UTC m=+65.146466318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle") pod "router-default-59b65c65d4-d5j44" (UID: "c53e1dae-b5f7-468f-a5d3-17cb726fcd33") : configmap references non-existent config key: service-ca.crt Sep 11 14:16:13.969353 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:13.969319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:13.969826 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:13.969360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:13.969826 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:13.969430 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:13.969826 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:13.969482 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:13.969826 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:13.969486 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm podName:b7958f39-de27-45ff-a3a3-cdc8f7908526 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:15.969471774 +0000 UTC m=+64.054928117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ddrcm" (UID: "b7958f39-de27-45ff-a3a3-cdc8f7908526") : configmap "kube-rbac-proxy" not found Sep 11 14:16:13.969826 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:13.969532 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls podName:b7958f39-de27-45ff-a3a3-cdc8f7908526 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:15.96952137 +0000 UTC m=+64.054977708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ddrcm" (UID: "b7958f39-de27-45ff-a3a3-cdc8f7908526") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:13.988815 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:13.988796 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2fj2j_a062adf8-6207-40d7-8eb0-302fede4913a/dns-node-resolver/0.log" Sep 11 14:16:14.790720 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:14.790691 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7plbw_d4dec9ea-724d-422e-8783-4795fe390f80/node-ca/0.log" Sep 11 14:16:15.986178 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:15.986141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:15.986178 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:15.986177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:15.986574 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:15.986269 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:15.986574 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:15.986332 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm podName:b7958f39-de27-45ff-a3a3-cdc8f7908526 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:19.986317877 +0000 UTC m=+68.071774215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ddrcm" (UID: "b7958f39-de27-45ff-a3a3-cdc8f7908526") : configmap "kube-rbac-proxy" not found Sep 11 14:16:15.986574 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:15.986356 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:15.986574 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:15.986403 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls podName:b7958f39-de27-45ff-a3a3-cdc8f7908526 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:19.986392032 +0000 UTC m=+68.071848371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ddrcm" (UID: "b7958f39-de27-45ff-a3a3-cdc8f7908526") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:15.989403 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:15.989383 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r5jzm_a30eb9bb-451d-4399-8180-10739866c4a4/serve-healthcheck-canary/0.log" Sep 11 14:16:16.993444 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:16.993400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:16.993845 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:16.993560 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 14:16:16.993845 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:16.993623 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls podName:9008fdf9-a87b-4914-8469-d60013b1f55d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:24.993609764 +0000 UTC m=+73.079066107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-47h9n" (UID: "9008fdf9-a87b-4914-8469-d60013b1f55d") : secret "cluster-monitoring-operator-tls" not found Sep 11 14:16:17.094517 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:17.094479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:17.094592 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:17.094536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-gt6s9\" (UID: \"be9ad885-960c-4e3e-b2c1-ecc75a2010c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:17.094592 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:17.094560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:17.094691 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:17.094622 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 14:16:17.094691 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:17.094684 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs podName:c53e1dae-b5f7-468f-a5d3-17cb726fcd33 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:25.094668711 +0000 UTC m=+73.180125049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs") pod "router-default-59b65c65d4-d5j44" (UID: "c53e1dae-b5f7-468f-a5d3-17cb726fcd33") : secret "router-metrics-certs-default" not found Sep 11 14:16:17.094817 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:17.094700 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle podName:c53e1dae-b5f7-468f-a5d3-17cb726fcd33 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:25.094689606 +0000 UTC m=+73.180145944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle") pod "router-default-59b65c65d4-d5j44" (UID: "c53e1dae-b5f7-468f-a5d3-17cb726fcd33") : configmap references non-existent config key: service-ca.crt Sep 11 14:16:17.094817 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:17.094708 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 14:16:17.094817 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:17.094788 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls podName:be9ad885-960c-4e3e-b2c1-ecc75a2010c7 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:25.094771186 +0000 UTC m=+73.180227527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-gt6s9" (UID: "be9ad885-960c-4e3e-b2c1-ecc75a2010c7") : secret "samples-operator-tls" not found Sep 11 14:16:18.102363 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.102328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:16:18.102363 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.102370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:16:18.102855 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.102553 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:16:18.102855 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:18.102712 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:16:18.102855 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:18.102801 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls podName:e9ae826c-e270-4612-9d92-9982ce13b392 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:50.102779844 +0000 UTC m=+98.188236181 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls") pod "dns-default-qdqt7" (UID: "e9ae826c-e270-4612-9d92-9982ce13b392") : secret "dns-default-metrics-tls" not found Sep 11 14:16:18.105212 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.105192 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 14:16:18.105414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.105398 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 14:16:18.112822 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:18.112803 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 14:16:18.112902 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:18.112847 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs podName:df1160f9-2ec0-4907-9ca2-4b269875d943 nodeName:}" failed. No retries permitted until 2026-09-11 14:17:22.112834702 +0000 UTC m=+130.198291045 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs") pod "network-metrics-daemon-pp2sq" (UID: "df1160f9-2ec0-4907-9ca2-4b269875d943") : secret "metrics-daemon-secret" not found Sep 11 14:16:18.114815 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.114797 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8-original-pull-secret\") pod \"global-pull-secret-syncer-crnx9\" (UID: \"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8\") " pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:16:18.203828 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.203785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7cj\" (UniqueName: \"kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj\") pod \"network-check-target-75q2d\" (UID: \"c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f\") " pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:16:18.206855 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.206839 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 14:16:18.218205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.218186 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 14:16:18.227070 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.227050 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh7cj\" (UniqueName: \"kubernetes.io/projected/c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f-kube-api-access-wh7cj\") pod \"network-check-target-75q2d\" (UID: \"c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f\") " pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:16:18.272885 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.272871 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-brmn7\"" Sep 11 14:16:18.280524 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.280506 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:16:18.367371 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.367299 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-crnx9" Sep 11 14:16:18.552605 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.552584 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-75q2d"] Sep 11 14:16:18.555098 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:18.555072 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ae9f92_66c3_49bd_bd0f_b4cd9962e58f.slice/crio-c409f18a5d7eec3498831ca0efca4cc31b8d10fd0a4f4e17368ab21dc655a301 WatchSource:0}: Error finding container c409f18a5d7eec3498831ca0efca4cc31b8d10fd0a4f4e17368ab21dc655a301: Status 404 returned error can't find the container with id c409f18a5d7eec3498831ca0efca4cc31b8d10fd0a4f4e17368ab21dc655a301 Sep 11 14:16:18.646052 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.645984 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-crnx9"] Sep 11 14:16:18.648980 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:18.648958 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33d7e0c_c13c_43fd_8cac_a0a5c8adc6e8.slice/crio-d95a714edc4979455342bca3393d809b3d51cdb6f74b49ef06692786bdde1061 WatchSource:0}: Error finding container d95a714edc4979455342bca3393d809b3d51cdb6f74b49ef06692786bdde1061: Status 404 returned error can't find the container with id d95a714edc4979455342bca3393d809b3d51cdb6f74b49ef06692786bdde1061 Sep 11 14:16:18.750653 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.750618 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-crnx9" event={"ID":"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8","Type":"ContainerStarted","Data":"d95a714edc4979455342bca3393d809b3d51cdb6f74b49ef06692786bdde1061"} Sep 11 14:16:18.751756 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:18.751716 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-75q2d" event={"ID":"c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f","Type":"ContainerStarted","Data":"c409f18a5d7eec3498831ca0efca4cc31b8d10fd0a4f4e17368ab21dc655a301"} Sep 11 14:16:20.017908 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:20.017347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:20.017908 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:20.017391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:20.017908 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:20.017505 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:20.017908 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:20.017551 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls podName:b7958f39-de27-45ff-a3a3-cdc8f7908526 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:28.017538715 +0000 UTC m=+76.102995053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ddrcm" (UID: "b7958f39-de27-45ff-a3a3-cdc8f7908526") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:20.017908 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:20.017592 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:20.017908 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:20.017616 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm podName:b7958f39-de27-45ff-a3a3-cdc8f7908526 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:28.01760694 +0000 UTC m=+76.103063279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ddrcm" (UID: "b7958f39-de27-45ff-a3a3-cdc8f7908526") : configmap "kube-rbac-proxy" not found Sep 11 14:16:23.768301 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:23.768264 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-crnx9" event={"ID":"b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8","Type":"ContainerStarted","Data":"8ccea212dee9ebeeb2995677bd621fd46577e0bb2945113ba38eff23cc0cdcc8"} Sep 11 14:16:23.769789 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:23.769768 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-75q2d" event={"ID":"c7ae9f92-66c3-49bd-bd0f-b4cd9962e58f","Type":"ContainerStarted","Data":"effe66e6f6c3adb8a23fd5053970cdd327169b2aee805a4dc918ecf18a178285"} Sep 11 14:16:23.769893 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:23.769829 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:16:23.793141 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:23.793098 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-crnx9" podStartSLOduration=67.166286447 podStartE2EDuration="1m11.793087471s" totalImagesPullingTime="4.626801024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:23.792063593 +0000 UTC m=+71.877519965" watchObservedRunningTime="2026-09-11 14:16:23.793087471 +0000 UTC m=+71.878543866" Sep 11 14:16:23.814049 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:23.814011 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-75q2d" podStartSLOduration=67.105072743 podStartE2EDuration="1m11.813998824s" totalImagesPullingTime="4.708926081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:23.813168223 +0000 UTC m=+71.898624581" watchObservedRunningTime="2026-09-11 14:16:23.813998824 +0000 UTC m=+71.899455234" Sep 11 14:16:25.057396 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:25.057349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:25.057864 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:25.057464 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 14:16:25.057864 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:25.057512 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls podName:9008fdf9-a87b-4914-8469-d60013b1f55d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:41.057500268 +0000 UTC m=+89.142956606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-47h9n" (UID: "9008fdf9-a87b-4914-8469-d60013b1f55d") : secret "cluster-monitoring-operator-tls" not found Sep 11 14:16:25.158326 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:25.158292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:25.158423 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:25.158408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:25.158473 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:25.158457 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle podName:c53e1dae-b5f7-468f-a5d3-17cb726fcd33 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:41.158439828 +0000 UTC m=+89.243896179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle") pod "router-default-59b65c65d4-d5j44" (UID: "c53e1dae-b5f7-468f-a5d3-17cb726fcd33") : configmap references non-existent config key: service-ca.crt Sep 11 14:16:25.158514 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:25.158489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-gt6s9\" (UID: \"be9ad885-960c-4e3e-b2c1-ecc75a2010c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:25.158555 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:25.158509 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 14:16:25.158555 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:25.158546 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs podName:c53e1dae-b5f7-468f-a5d3-17cb726fcd33 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:41.158534882 +0000 UTC m=+89.243991220 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs") pod "router-default-59b65c65d4-d5j44" (UID: "c53e1dae-b5f7-468f-a5d3-17cb726fcd33") : secret "router-metrics-certs-default" not found Sep 11 14:16:25.158632 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:25.158562 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 14:16:25.158632 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:25.158611 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls podName:be9ad885-960c-4e3e-b2c1-ecc75a2010c7 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:41.15859913 +0000 UTC m=+89.244055468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-gt6s9" (UID: "be9ad885-960c-4e3e-b2c1-ecc75a2010c7") : secret "samples-operator-tls" not found Sep 11 14:16:28.079962 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:28.079913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:28.079962 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:28.079965 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:28.080368 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:28.080051 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:28.080368 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:28.080108 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls podName:b7958f39-de27-45ff-a3a3-cdc8f7908526 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:44.080093527 +0000 UTC m=+92.165549868 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ddrcm" (UID: "b7958f39-de27-45ff-a3a3-cdc8f7908526") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:28.080457 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:28.080436 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7958f39-de27-45ff-a3a3-cdc8f7908526-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:41.075657 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.075515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:41.077999 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.077974 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9008fdf9-a87b-4914-8469-d60013b1f55d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-47h9n\" (UID: \"9008fdf9-a87b-4914-8469-d60013b1f55d\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:41.175988 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.175945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:41.175988 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.175994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-gt6s9\" (UID: \"be9ad885-960c-4e3e-b2c1-ecc75a2010c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:41.176174 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.176025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:41.176673 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.176651 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-service-ca-bundle\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:41.178280 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.178250 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c53e1dae-b5f7-468f-a5d3-17cb726fcd33-metrics-certs\") pod \"router-default-59b65c65d4-d5j44\" (UID: \"c53e1dae-b5f7-468f-a5d3-17cb726fcd33\") " pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:41.178280 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.178275 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be9ad885-960c-4e3e-b2c1-ecc75a2010c7-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-gt6s9\" (UID: \"be9ad885-960c-4e3e-b2c1-ecc75a2010c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:41.332338 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.332271 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pxtph\"" Sep 11 14:16:41.336308 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.336293 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" Sep 11 14:16:41.438250 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.438227 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jdpqx\"" Sep 11 14:16:41.444863 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.444846 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" Sep 11 14:16:41.450481 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.450466 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mgqlq\"" Sep 11 14:16:41.457817 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.457788 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:41.621121 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.621093 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n"] Sep 11 14:16:41.623850 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:41.623827 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9008fdf9_a87b_4914_8469_d60013b1f55d.slice/crio-7c615045c20a2d8622abfc2327e753b1d7841a7ce47b2d4c2abd29d9b4cd0620 WatchSource:0}: Error finding container 7c615045c20a2d8622abfc2327e753b1d7841a7ce47b2d4c2abd29d9b4cd0620: Status 404 returned error can't find the container with id 7c615045c20a2d8622abfc2327e753b1d7841a7ce47b2d4c2abd29d9b4cd0620 Sep 11 14:16:41.722540 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.722520 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9"] Sep 11 14:16:41.745452 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.745431 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59b65c65d4-d5j44"] Sep 11 14:16:41.748063 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:41.748042 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53e1dae_b5f7_468f_a5d3_17cb726fcd33.slice/crio-f9a4cc6d21aab7ff81513d8ee08866d2778ff65b22c2952590aeee07798a99e4 WatchSource:0}: Error finding container f9a4cc6d21aab7ff81513d8ee08866d2778ff65b22c2952590aeee07798a99e4: Status 404 returned error can't find the container with id f9a4cc6d21aab7ff81513d8ee08866d2778ff65b22c2952590aeee07798a99e4 Sep 11 14:16:41.818878 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.818843 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" event={"ID":"be9ad885-960c-4e3e-b2c1-ecc75a2010c7","Type":"ContainerStarted","Data":"f9d4ba3afb4924961aafe66849cb9a72019bb2f564407ac7def7e77e4580c407"} Sep 11 14:16:41.820315 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.820282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" event={"ID":"9008fdf9-a87b-4914-8469-d60013b1f55d","Type":"ContainerStarted","Data":"7c615045c20a2d8622abfc2327e753b1d7841a7ce47b2d4c2abd29d9b4cd0620"} Sep 11 14:16:41.821974 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.821950 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59b65c65d4-d5j44" event={"ID":"c53e1dae-b5f7-468f-a5d3-17cb726fcd33","Type":"ContainerStarted","Data":"9a94c43d5a23852fa9ac1e43956e1c93012b15eb3c29230ba7893ddee3b890ea"} Sep 11 14:16:41.822065 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.821992 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59b65c65d4-d5j44" event={"ID":"c53e1dae-b5f7-468f-a5d3-17cb726fcd33","Type":"ContainerStarted","Data":"f9a4cc6d21aab7ff81513d8ee08866d2778ff65b22c2952590aeee07798a99e4"} Sep 11 14:16:41.857468 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:41.857425 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-59b65c65d4-d5j44" podStartSLOduration=32.857412051 podStartE2EDuration="32.857412051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:41.856411041 +0000 UTC m=+89.941867401" watchObservedRunningTime="2026-09-11 14:16:41.857412051 +0000 UTC m=+89.942868405" Sep 11 14:16:42.463310 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:42.463283 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:42.463711 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:42.463378 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:42.825410 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:42.825325 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:42.826571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:42.826545 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-59b65c65d4-d5j44" Sep 11 14:16:43.830134 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:43.830085 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" event={"ID":"be9ad885-960c-4e3e-b2c1-ecc75a2010c7","Type":"ContainerStarted","Data":"cc11134b0f4c8e16243ba8e31ac9d1130062c06913c484d0af055ed71641770a"} Sep 11 14:16:43.830570 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:43.830176 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" event={"ID":"be9ad885-960c-4e3e-b2c1-ecc75a2010c7","Type":"ContainerStarted","Data":"5459f32cefff307c69e72110cab79ae651f3a5dac1b8ece464799f4a9af5235d"} Sep 11 14:16:43.831723 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:43.831691 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" event={"ID":"9008fdf9-a87b-4914-8469-d60013b1f55d","Type":"ContainerStarted","Data":"b2820a081a1b8519d9672b7fee61582c16cc75067ac9a74109fd320bcc70f038"} Sep 11 14:16:43.867087 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:43.867043 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-gt6s9" podStartSLOduration=32.996925589 podStartE2EDuration="34.867029812s" totalImagesPullingTime="1.870104223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:43.866074579 +0000 UTC m=+91.951530944" watchObservedRunningTime="2026-09-11 14:16:43.867029812 +0000 UTC m=+91.952486172" Sep 11 14:16:43.889436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:43.889356 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-47h9n" podStartSLOduration=32.880264962 podStartE2EDuration="34.889343293s" totalImagesPullingTime="2.009078331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:43.888572829 +0000 UTC m=+91.974029190" watchObservedRunningTime="2026-09-11 14:16:43.889343293 +0000 UTC m=+91.974799682" Sep 11 14:16:44.096415 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:44.096379 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:44.098999 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:44.098967 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7958f39-de27-45ff-a3a3-cdc8f7908526-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ddrcm\" (UID: \"b7958f39-de27-45ff-a3a3-cdc8f7908526\") " pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:44.359349 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:44.359320 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ddrcm" Sep 11 14:16:44.647496 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:44.647472 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ddrcm"] Sep 11 14:16:44.649119 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:44.649091 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7958f39_de27_45ff_a3a3_cdc8f7908526.slice/crio-02656190b8f93452d50fe3dc3d1b573d1f7e520ae49b6658b307d09e769b45dd WatchSource:0}: Error finding container 02656190b8f93452d50fe3dc3d1b573d1f7e520ae49b6658b307d09e769b45dd: Status 404 returned error can't find the container with id 02656190b8f93452d50fe3dc3d1b573d1f7e520ae49b6658b307d09e769b45dd Sep 11 14:16:44.836393 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:44.836356 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ddrcm" event={"ID":"b7958f39-de27-45ff-a3a3-cdc8f7908526","Type":"ContainerStarted","Data":"32205e260ce5cf326f0d4c11fde05ea8e84e05167052552f6249397f9a5dd2ee"} Sep 11 14:16:44.836853 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:44.836426 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ddrcm" event={"ID":"b7958f39-de27-45ff-a3a3-cdc8f7908526","Type":"ContainerStarted","Data":"02656190b8f93452d50fe3dc3d1b573d1f7e520ae49b6658b307d09e769b45dd"} Sep 11 14:16:45.842525 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:45.842489 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ddrcm" event={"ID":"b7958f39-de27-45ff-a3a3-cdc8f7908526","Type":"ContainerStarted","Data":"242695b02d4918a323901616387ce8ad566edd81456390283539b7c88a556baf"} Sep 11 14:16:46.847374 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:46.847295 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ddrcm" event={"ID":"b7958f39-de27-45ff-a3a3-cdc8f7908526","Type":"ContainerStarted","Data":"2de6b0f1d5dd602a4ccb10bb1f72c64f4574676f532217f1f7a6a9cb93a29b3d"} Sep 11 14:16:46.903496 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:46.903449 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ddrcm" podStartSLOduration=33.152694185 podStartE2EDuration="34.903435324s" totalImagesPullingTime="1.750741139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:16:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:46.903056402 +0000 UTC m=+94.988512767" watchObservedRunningTime="2026-09-11 14:16:46.903435324 +0000 UTC m=+94.988891683" Sep 11 14:16:47.133276 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.133240 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj"] Sep 11 14:16:47.136492 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.136467 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj" Sep 11 14:16:47.152085 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.152061 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 11 14:16:47.152085 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.152061 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-w29x2\"" Sep 11 14:16:47.152264 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.152097 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 11 14:16:47.215852 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.215825 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj"] Sep 11 14:16:47.222211 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.222185 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd588e0d-e6f8-4ffc-9a47-5a5d625514fd-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-gp6gj\" (UID: \"dd588e0d-e6f8-4ffc-9a47-5a5d625514fd\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj" Sep 11 14:16:47.222316 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.222243 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd588e0d-e6f8-4ffc-9a47-5a5d625514fd-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-gp6gj\" (UID: \"dd588e0d-e6f8-4ffc-9a47-5a5d625514fd\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj" Sep 11 14:16:47.322745 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.322695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd588e0d-e6f8-4ffc-9a47-5a5d625514fd-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-gp6gj\" (UID: \"dd588e0d-e6f8-4ffc-9a47-5a5d625514fd\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj" Sep 11 14:16:47.322861 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.322760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd588e0d-e6f8-4ffc-9a47-5a5d625514fd-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-gp6gj\" (UID: \"dd588e0d-e6f8-4ffc-9a47-5a5d625514fd\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj" Sep 11 14:16:47.323331 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.323313 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd588e0d-e6f8-4ffc-9a47-5a5d625514fd-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-gp6gj\" (UID: \"dd588e0d-e6f8-4ffc-9a47-5a5d625514fd\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj" Sep 11 14:16:47.324992 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.324973 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd588e0d-e6f8-4ffc-9a47-5a5d625514fd-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-gp6gj\" (UID: \"dd588e0d-e6f8-4ffc-9a47-5a5d625514fd\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj" Sep 11 14:16:47.360905 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.360879 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg"] Sep 11 14:16:47.363853 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.363837 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48"] Sep 11 14:16:47.363979 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.363963 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg" Sep 11 14:16:47.366521 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.366502 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48" Sep 11 14:16:47.378000 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:47.377974 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"prometheus-operator-admission-webhook-tls\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret" Sep 11 14:16:47.379558 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:47.379538 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 11 14:16:47.380557 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:47.380533 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"managed-serviceaccount-addon-agent-6f979db9f6-twzxg\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="6579dace-ad76-4eb3-b43b-e7df05d7f085" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg" Sep 11 14:16:47.380773 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:47.380753 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 11 14:16:47.381864 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:47.381845 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-dockercfg-fqv4r\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-fqv4r\"" type="*v1.Secret" Sep 11 14:16:47.387593 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:47.387572 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" Sep 11 14:16:47.387679 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:47.387655 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" type="*v1.Secret" Sep 11 14:16:47.393275 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:47.393255 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"prometheus-operator-admission-webhook-5864cd99d7-qld48\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="9584ca71-fb39-4093-9237-189e287eccc5" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48" Sep 11 14:16:47.393329 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:47.393276 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"prometheus-operator-admission-webhook-dockercfg-xgjsd\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xgjsd\"" type="*v1.Secret" Sep 11 14:16:47.396035 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:47.396015 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"managed-serviceaccount-addon-agent-6f979db9f6-twzxg\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="6579dace-ad76-4eb3-b43b-e7df05d7f085" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg" Sep 11 14:16:47.423449 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.423418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxvb6\" (UniqueName: \"kubernetes.io/projected/6579dace-ad76-4eb3-b43b-e7df05d7f085-kube-api-access-zxvb6\") pod \"managed-serviceaccount-addon-agent-6f979db9f6-twzxg\" (UID: \"6579dace-ad76-4eb3-b43b-e7df05d7f085\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg" Sep 11 14:16:47.423522 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.423476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9584ca71-fb39-4093-9237-189e287eccc5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qld48\" (UID: \"9584ca71-fb39-4093-9237-189e287eccc5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48" Sep 11 14:16:47.423522 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.423499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6579dace-ad76-4eb3-b43b-e7df05d7f085-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f979db9f6-twzxg\" (UID: \"6579dace-ad76-4eb3-b43b-e7df05d7f085\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg" Sep 11 14:16:47.445509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.445491 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj" Sep 11 14:16:47.447710 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.447689 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48"] Sep 11 14:16:47.448795 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.448773 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg"] Sep 11 14:16:47.524634 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.524032 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxvb6\" (UniqueName: \"kubernetes.io/projected/6579dace-ad76-4eb3-b43b-e7df05d7f085-kube-api-access-zxvb6\") pod \"managed-serviceaccount-addon-agent-6f979db9f6-twzxg\" (UID: \"6579dace-ad76-4eb3-b43b-e7df05d7f085\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg" Sep 11 14:16:47.524634 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.524116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9584ca71-fb39-4093-9237-189e287eccc5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qld48\" (UID: \"9584ca71-fb39-4093-9237-189e287eccc5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48" Sep 11 14:16:47.524634 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.524145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6579dace-ad76-4eb3-b43b-e7df05d7f085-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f979db9f6-twzxg\" (UID: \"6579dace-ad76-4eb3-b43b-e7df05d7f085\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg" Sep 11 14:16:47.748007 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.747981 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj"] Sep 11 14:16:47.750165 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:47.750133 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd588e0d_e6f8_4ffc_9a47_5a5d625514fd.slice/crio-d180bca92b2b3397a1aad540b00137912d8d5d5bff515705dd008e8362f61246 WatchSource:0}: Error finding container d180bca92b2b3397a1aad540b00137912d8d5d5bff515705dd008e8362f61246: Status 404 returned error can't find the container with id d180bca92b2b3397a1aad540b00137912d8d5d5bff515705dd008e8362f61246 Sep 11 14:16:47.851014 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:47.850984 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj" event={"ID":"dd588e0d-e6f8-4ffc-9a47-5a5d625514fd","Type":"ContainerStarted","Data":"d180bca92b2b3397a1aad540b00137912d8d5d5bff515705dd008e8362f61246"} Sep 11 14:16:48.281980 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:48.281949 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 14:16:48.298846 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:48.298822 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 14:16:48.306309 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:48.306283 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 11 14:16:48.317166 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:48.317144 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6579dace-ad76-4eb3-b43b-e7df05d7f085-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f979db9f6-twzxg\" (UID: \"6579dace-ad76-4eb3-b43b-e7df05d7f085\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg" Sep 11 14:16:48.390039 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:48.390012 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-fqv4r\"" Sep 11 14:16:48.525556 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:48.525503 2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: failed to sync secret cache: timed out waiting for the condition Sep 11 14:16:48.525745 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:48.525590 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9584ca71-fb39-4093-9237-189e287eccc5-tls-certificates podName:9584ca71-fb39-4093-9237-189e287eccc5 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:49.025569716 +0000 UTC m=+97.111026054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/9584ca71-fb39-4093-9237-189e287eccc5-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-qld48" (UID: "9584ca71-fb39-4093-9237-189e287eccc5") : failed to sync secret cache: timed out waiting for the condition Sep 11 14:16:48.623244 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:48.623216 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xgjsd\"" Sep 11 14:16:48.820381 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:48.820356 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 11 14:16:48.855151 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:48.855121 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj" event={"ID":"dd588e0d-e6f8-4ffc-9a47-5a5d625514fd","Type":"ContainerStarted","Data":"b95d84790eba2804857562f7b6cbbc93d77ce863983e75d0174884b21fb8afa2"} Sep 11 14:16:48.890535 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:48.890481 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 14:16:48.899414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:48.899385 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxvb6\" (UniqueName: \"kubernetes.io/projected/6579dace-ad76-4eb3-b43b-e7df05d7f085-kube-api-access-zxvb6\") pod \"managed-serviceaccount-addon-agent-6f979db9f6-twzxg\" (UID: \"6579dace-ad76-4eb3-b43b-e7df05d7f085\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg" Sep 11 14:16:48.979854 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:48.979816 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-gp6gj" podStartSLOduration=0.97236211 podStartE2EDuration="1.979800975s" totalImagesPullingTime="1.007438865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:48.935536883 +0000 UTC m=+97.020993242" watchObservedRunningTime="2026-09-11 14:16:48.979800975 +0000 UTC m=+97.065257334" Sep 11 14:16:49.036116 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:49.036086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9584ca71-fb39-4093-9237-189e287eccc5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qld48\" (UID: \"9584ca71-fb39-4093-9237-189e287eccc5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48" Sep 11 14:16:49.038271 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:49.038250 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9584ca71-fb39-4093-9237-189e287eccc5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qld48\" (UID: \"9584ca71-fb39-4093-9237-189e287eccc5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48" Sep 11 14:16:49.183265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:49.183214 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg" Sep 11 14:16:49.188076 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:49.188063 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48" Sep 11 14:16:49.485298 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:49.485189 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg"] Sep 11 14:16:49.486023 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:49.486000 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48"] Sep 11 14:16:49.487155 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:49.487131 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6579dace_ad76_4eb3_b43b_e7df05d7f085.slice/crio-917392f109d5d72ef50244bf405a8e16e2fb7e0ac5bf9e8f6186da02ebb1fc2b WatchSource:0}: Error finding container 917392f109d5d72ef50244bf405a8e16e2fb7e0ac5bf9e8f6186da02ebb1fc2b: Status 404 returned error can't find the container with id 917392f109d5d72ef50244bf405a8e16e2fb7e0ac5bf9e8f6186da02ebb1fc2b Sep 11 14:16:49.487493 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:49.487476 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9584ca71_fb39_4093_9237_189e287eccc5.slice/crio-28382a7655b0194af3556b4abf02b43f58e80bed40420acb05466cc20c9e722a WatchSource:0}: Error finding container 28382a7655b0194af3556b4abf02b43f58e80bed40420acb05466cc20c9e722a: Status 404 returned error can't find the container with id 28382a7655b0194af3556b4abf02b43f58e80bed40420acb05466cc20c9e722a Sep 11 14:16:49.859050 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:49.859012 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg" event={"ID":"6579dace-ad76-4eb3-b43b-e7df05d7f085","Type":"ContainerStarted","Data":"917392f109d5d72ef50244bf405a8e16e2fb7e0ac5bf9e8f6186da02ebb1fc2b"} Sep 11 14:16:49.860086 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:49.860061 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48" event={"ID":"9584ca71-fb39-4093-9237-189e287eccc5","Type":"ContainerStarted","Data":"28382a7655b0194af3556b4abf02b43f58e80bed40420acb05466cc20c9e722a"} Sep 11 14:16:50.145074 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:50.144987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:16:50.148046 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:50.148017 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ae826c-e270-4612-9d92-9982ce13b392-metrics-tls\") pod \"dns-default-qdqt7\" (UID: \"e9ae826c-e270-4612-9d92-9982ce13b392\") " pod="openshift-dns/dns-default-qdqt7" Sep 11 14:16:50.228389 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:50.228190 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c4f62\"" Sep 11 14:16:50.235847 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:50.235821 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qdqt7" Sep 11 14:16:50.743194 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:50.743118 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qdqt7"] Sep 11 14:16:50.746955 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:50.746931 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ae826c_e270_4612_9d92_9982ce13b392.slice/crio-7442e3d88b7b4701862246a20234e4e7b47c06991c444f626d8cdfa15d9fbe24 WatchSource:0}: Error finding container 7442e3d88b7b4701862246a20234e4e7b47c06991c444f626d8cdfa15d9fbe24: Status 404 returned error can't find the container with id 7442e3d88b7b4701862246a20234e4e7b47c06991c444f626d8cdfa15d9fbe24 Sep 11 14:16:50.865471 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:50.865431 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdqt7" event={"ID":"e9ae826c-e270-4612-9d92-9982ce13b392","Type":"ContainerStarted","Data":"7442e3d88b7b4701862246a20234e4e7b47c06991c444f626d8cdfa15d9fbe24"} Sep 11 14:16:50.867144 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:50.867113 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48" event={"ID":"9584ca71-fb39-4093-9237-189e287eccc5","Type":"ContainerStarted","Data":"4eb90e114dc1c3f09c7f633c6b103306c161d24d1a81aa41a49600a8b9ff068e"} Sep 11 14:16:50.867349 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:50.867322 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48" Sep 11 14:16:50.872667 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:50.872645 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48" Sep 11 14:16:50.895016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:50.894973 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qld48" podStartSLOduration=2.8636767389999997 podStartE2EDuration="3.894959101s" totalImagesPullingTime="1.031282362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:50.894748439 +0000 UTC m=+98.980204797" watchObservedRunningTime="2026-09-11 14:16:50.894959101 +0000 UTC m=+98.980415461" Sep 11 14:16:51.421465 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.421428 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-s7scw"] Sep 11 14:16:51.433839 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.433815 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:51.436857 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.436833 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 11 14:16:51.438013 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.437990 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-44f7b\"" Sep 11 14:16:51.438013 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.438003 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 11 14:16:51.438163 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.438049 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 14:16:51.444597 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.444576 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-s7scw"] Sep 11 14:16:51.556836 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.556799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fabcc40-b774-42df-b33e-021b8b35650d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-s7scw\" (UID: \"8fabcc40-b774-42df-b33e-021b8b35650d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:51.557025 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.556877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fabcc40-b774-42df-b33e-021b8b35650d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-s7scw\" (UID: \"8fabcc40-b774-42df-b33e-021b8b35650d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:51.557025 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.556944 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt9d5\" (UniqueName: \"kubernetes.io/projected/8fabcc40-b774-42df-b33e-021b8b35650d-kube-api-access-xt9d5\") pod \"prometheus-operator-68bcd7f987-s7scw\" (UID: \"8fabcc40-b774-42df-b33e-021b8b35650d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:51.557025 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.556995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fabcc40-b774-42df-b33e-021b8b35650d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-s7scw\" (UID: \"8fabcc40-b774-42df-b33e-021b8b35650d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:51.658145 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.658109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fabcc40-b774-42df-b33e-021b8b35650d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-s7scw\" (UID: \"8fabcc40-b774-42df-b33e-021b8b35650d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:51.658318 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.658157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fabcc40-b774-42df-b33e-021b8b35650d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-s7scw\" (UID: \"8fabcc40-b774-42df-b33e-021b8b35650d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:51.658318 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.658180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt9d5\" (UniqueName: \"kubernetes.io/projected/8fabcc40-b774-42df-b33e-021b8b35650d-kube-api-access-xt9d5\") pod \"prometheus-operator-68bcd7f987-s7scw\" (UID: \"8fabcc40-b774-42df-b33e-021b8b35650d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:51.658318 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.658213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fabcc40-b774-42df-b33e-021b8b35650d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-s7scw\" (UID: \"8fabcc40-b774-42df-b33e-021b8b35650d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:51.658990 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.658950 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fabcc40-b774-42df-b33e-021b8b35650d-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-s7scw\" (UID: \"8fabcc40-b774-42df-b33e-021b8b35650d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:51.661035 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.660988 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fabcc40-b774-42df-b33e-021b8b35650d-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-s7scw\" (UID: \"8fabcc40-b774-42df-b33e-021b8b35650d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:51.661149 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.661070 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fabcc40-b774-42df-b33e-021b8b35650d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-s7scw\" (UID: \"8fabcc40-b774-42df-b33e-021b8b35650d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:51.672512 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.672435 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt9d5\" (UniqueName: \"kubernetes.io/projected/8fabcc40-b774-42df-b33e-021b8b35650d-kube-api-access-xt9d5\") pod \"prometheus-operator-68bcd7f987-s7scw\" (UID: \"8fabcc40-b774-42df-b33e-021b8b35650d\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:51.743343 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:51.743317 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" Sep 11 14:16:52.072475 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:52.072394 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-s7scw"] Sep 11 14:16:52.076273 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:52.076250 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fabcc40_b774_42df_b33e_021b8b35650d.slice/crio-b13e8a79940ff567f3d6b5f4b0f3438a4bec6140e255af5ce2e5080f9a922cb5 WatchSource:0}: Error finding container b13e8a79940ff567f3d6b5f4b0f3438a4bec6140e255af5ce2e5080f9a922cb5: Status 404 returned error can't find the container with id b13e8a79940ff567f3d6b5f4b0f3438a4bec6140e255af5ce2e5080f9a922cb5 Sep 11 14:16:52.876475 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:52.876434 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg" event={"ID":"6579dace-ad76-4eb3-b43b-e7df05d7f085","Type":"ContainerStarted","Data":"59fa5ec8e17f93b878525caa38c5c367e4e1be309abe4f78b7b500bd519e1001"} Sep 11 14:16:52.878032 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:52.877997 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" event={"ID":"8fabcc40-b774-42df-b33e-021b8b35650d","Type":"ContainerStarted","Data":"b13e8a79940ff567f3d6b5f4b0f3438a4bec6140e255af5ce2e5080f9a922cb5"} Sep 11 14:16:52.899484 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:52.899440 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f979db9f6-twzxg" podStartSLOduration=3.52672008 podStartE2EDuration="5.899426909s" totalImagesPullingTime="2.372706829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:52.898401208 +0000 UTC m=+100.983857568" watchObservedRunningTime="2026-09-11 14:16:52.899426909 +0000 UTC m=+100.984883268" Sep 11 14:16:53.882671 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:53.882628 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdqt7" event={"ID":"e9ae826c-e270-4612-9d92-9982ce13b392","Type":"ContainerStarted","Data":"378c93365e81e8a5754d33ec365e88e0713705aaabcf405e0d1947fe4d4eaabf"} Sep 11 14:16:53.883073 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:53.882694 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qdqt7" Sep 11 14:16:53.883073 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:53.882719 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdqt7" event={"ID":"e9ae826c-e270-4612-9d92-9982ce13b392","Type":"ContainerStarted","Data":"9b9a1a4744e451d391f27288f2f2528e770ec61b291fbee367aaf2989571979d"} Sep 11 14:16:53.884417 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:53.884391 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" event={"ID":"8fabcc40-b774-42df-b33e-021b8b35650d","Type":"ContainerStarted","Data":"7b2e9b077910b823299c98a20a04fd91582dc1a1b47fa321d76a0a88e2e93d92"} Sep 11 14:16:53.884543 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:53.884431 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" event={"ID":"8fabcc40-b774-42df-b33e-021b8b35650d","Type":"ContainerStarted","Data":"1419ecb78e37d8ab38c6b9a1c457cd955cb0100cf585d67411cffe4403092e5d"} Sep 11 14:16:53.906554 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:53.906517 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qdqt7" podStartSLOduration=65.349370901 podStartE2EDuration="1m7.906503705s" totalImagesPullingTime="2.557132804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:53.904659744 +0000 UTC m=+101.990116105" watchObservedRunningTime="2026-09-11 14:16:53.906503705 +0000 UTC m=+101.991960064" Sep 11 14:16:53.926068 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:53.926031 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7scw" podStartSLOduration=1.6979113099999998 podStartE2EDuration="2.926019052s" totalImagesPullingTime="1.228107742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:53.925176399 +0000 UTC m=+102.010632758" watchObservedRunningTime="2026-09-11 14:16:53.926019052 +0000 UTC m=+102.011475412" Sep 11 14:16:54.775519 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:54.775487 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-75q2d" Sep 11 14:16:55.612063 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:55.612028 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-pkjx9"] Sep 11 14:16:55.632553 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:55.632521 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-pkjx9" Sep 11 14:16:55.633579 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:55.633559 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-pkjx9"] Sep 11 14:16:55.636623 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:55.636593 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 14:16:55.636905 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:55.636851 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 14:16:55.637037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:55.637016 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7sg52\"" Sep 11 14:16:55.687941 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:55.687917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chr8k\" (UniqueName: \"kubernetes.io/projected/859033b0-6de0-484b-b73c-eaea2affa856-kube-api-access-chr8k\") pod \"downloads-76f99584f4-pkjx9\" (UID: \"859033b0-6de0-484b-b73c-eaea2affa856\") " pod="openshift-console/downloads-76f99584f4-pkjx9" Sep 11 14:16:55.789212 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:55.789183 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chr8k\" (UniqueName: \"kubernetes.io/projected/859033b0-6de0-484b-b73c-eaea2affa856-kube-api-access-chr8k\") pod \"downloads-76f99584f4-pkjx9\" (UID: \"859033b0-6de0-484b-b73c-eaea2affa856\") " pod="openshift-console/downloads-76f99584f4-pkjx9" Sep 11 14:16:55.802664 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:55.802639 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chr8k\" (UniqueName: \"kubernetes.io/projected/859033b0-6de0-484b-b73c-eaea2affa856-kube-api-access-chr8k\") pod \"downloads-76f99584f4-pkjx9\" (UID: \"859033b0-6de0-484b-b73c-eaea2affa856\") " pod="openshift-console/downloads-76f99584f4-pkjx9" Sep 11 14:16:55.941450 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:55.941381 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-pkjx9" Sep 11 14:16:55.987626 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:55.987596 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz"] Sep 11 14:16:56.008566 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.008506 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz"] Sep 11 14:16:56.008705 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.008630 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.011591 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.011554 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-g9j7n\"" Sep 11 14:16:56.011777 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.011650 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 11 14:16:56.011860 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.011844 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 11 14:16:56.024116 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.023856 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nz7g9"] Sep 11 14:16:56.039158 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.039139 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s"] Sep 11 14:16:56.039286 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.039272 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.042687 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.042665 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 14:16:56.042819 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.042673 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 14:16:56.042819 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.042703 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 14:16:56.042989 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.042973 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qj5r6\"" Sep 11 14:16:56.053722 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.053703 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s"] Sep 11 14:16:56.053859 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.053812 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.056756 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.056683 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-f7zlc\"" Sep 11 14:16:56.056756 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.056711 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 14:16:56.056756 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.056713 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 14:16:56.056957 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.056785 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 14:16:56.090960 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.090937 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4050a332-4649-4891-ac88-36a83d70b6ba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.091066 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.090969 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4050a332-4649-4891-ac88-36a83d70b6ba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.091066 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.090989 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4050a332-4649-4891-ac88-36a83d70b6ba-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.091066 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/558b6beb-1341-417b-9763-85053d4d8b26-sys\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.091066 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091046 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/558b6beb-1341-417b-9763-85053d4d8b26-root\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.091254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091073 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75503659-d535-4d5b-9485-cd92d3a6cf09-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-h8wbz\" (UID: \"75503659-d535-4d5b-9485-cd92d3a6cf09\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.091254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trf87\" (UniqueName: \"kubernetes.io/projected/558b6beb-1341-417b-9763-85053d4d8b26-kube-api-access-trf87\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.091254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091172 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-textfile\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.091254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/558b6beb-1341-417b-9763-85053d4d8b26-metrics-client-ca\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.091254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-accelerators-collector-config\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.091528 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4050a332-4649-4891-ac88-36a83d70b6ba-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.091528 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-tls\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.091528 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb8b2\" (UniqueName: \"kubernetes.io/projected/75503659-d535-4d5b-9485-cd92d3a6cf09-kube-api-access-sb8b2\") pod \"openshift-state-metrics-67dcf8c979-h8wbz\" (UID: \"75503659-d535-4d5b-9485-cd92d3a6cf09\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.091528 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091409 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.091528 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4050a332-4649-4891-ac88-36a83d70b6ba-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.091528 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhz6h\" (UniqueName: \"kubernetes.io/projected/4050a332-4649-4891-ac88-36a83d70b6ba-kube-api-access-vhz6h\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.091528 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091488 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75503659-d535-4d5b-9485-cd92d3a6cf09-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-h8wbz\" (UID: \"75503659-d535-4d5b-9485-cd92d3a6cf09\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.091767 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091534 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-wtmp\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.091767 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.091564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75503659-d535-4d5b-9485-cd92d3a6cf09-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-h8wbz\" (UID: \"75503659-d535-4d5b-9485-cd92d3a6cf09\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.192106 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4050a332-4649-4891-ac88-36a83d70b6ba-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.192106 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/558b6beb-1341-417b-9763-85053d4d8b26-sys\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.192106 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/558b6beb-1341-417b-9763-85053d4d8b26-root\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.192106 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75503659-d535-4d5b-9485-cd92d3a6cf09-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-h8wbz\" (UID: \"75503659-d535-4d5b-9485-cd92d3a6cf09\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.192414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trf87\" (UniqueName: \"kubernetes.io/projected/558b6beb-1341-417b-9763-85053d4d8b26-kube-api-access-trf87\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.192414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-textfile\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.192414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192160 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/558b6beb-1341-417b-9763-85053d4d8b26-root\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.192414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192186 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/558b6beb-1341-417b-9763-85053d4d8b26-metrics-client-ca\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.192414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-accelerators-collector-config\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.192414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192160 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/558b6beb-1341-417b-9763-85053d4d8b26-sys\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.192414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4050a332-4649-4891-ac88-36a83d70b6ba-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.192414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-tls\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.192414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb8b2\" (UniqueName: \"kubernetes.io/projected/75503659-d535-4d5b-9485-cd92d3a6cf09-kube-api-access-sb8b2\") pod \"openshift-state-metrics-67dcf8c979-h8wbz\" (UID: \"75503659-d535-4d5b-9485-cd92d3a6cf09\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.192414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.192414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4050a332-4649-4891-ac88-36a83d70b6ba-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.192414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhz6h\" (UniqueName: \"kubernetes.io/projected/4050a332-4649-4891-ac88-36a83d70b6ba-kube-api-access-vhz6h\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.192838 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75503659-d535-4d5b-9485-cd92d3a6cf09-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-h8wbz\" (UID: \"75503659-d535-4d5b-9485-cd92d3a6cf09\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.192838 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192470 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-textfile\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.192838 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-wtmp\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.192838 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75503659-d535-4d5b-9485-cd92d3a6cf09-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-h8wbz\" (UID: \"75503659-d535-4d5b-9485-cd92d3a6cf09\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.192838 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4050a332-4649-4891-ac88-36a83d70b6ba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.192838 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.192601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4050a332-4649-4891-ac88-36a83d70b6ba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.193059 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.193020 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4050a332-4649-4891-ac88-36a83d70b6ba-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.193296 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.193269 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4050a332-4649-4891-ac88-36a83d70b6ba-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.193449 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.193425 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-accelerators-collector-config\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.193524 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.193506 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/558b6beb-1341-417b-9763-85053d4d8b26-metrics-client-ca\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.193594 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.193575 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-wtmp\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.193745 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.193706 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75503659-d535-4d5b-9485-cd92d3a6cf09-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-h8wbz\" (UID: \"75503659-d535-4d5b-9485-cd92d3a6cf09\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.194159 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.193831 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4050a332-4649-4891-ac88-36a83d70b6ba-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.195357 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.195322 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75503659-d535-4d5b-9485-cd92d3a6cf09-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-h8wbz\" (UID: \"75503659-d535-4d5b-9485-cd92d3a6cf09\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.195469 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.195447 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75503659-d535-4d5b-9485-cd92d3a6cf09-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-h8wbz\" (UID: \"75503659-d535-4d5b-9485-cd92d3a6cf09\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.195686 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.195656 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-tls\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.195789 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.195662 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4050a332-4649-4891-ac88-36a83d70b6ba-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.196004 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.195985 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4050a332-4649-4891-ac88-36a83d70b6ba-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.196139 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.196120 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/558b6beb-1341-417b-9763-85053d4d8b26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.207039 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.207018 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhz6h\" (UniqueName: \"kubernetes.io/projected/4050a332-4649-4891-ac88-36a83d70b6ba-kube-api-access-vhz6h\") pod \"kube-state-metrics-59fbdd77d-lzz8s\" (UID: \"4050a332-4649-4891-ac88-36a83d70b6ba\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.207852 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.207830 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trf87\" (UniqueName: \"kubernetes.io/projected/558b6beb-1341-417b-9763-85053d4d8b26-kube-api-access-trf87\") pod \"node-exporter-nz7g9\" (UID: \"558b6beb-1341-417b-9763-85053d4d8b26\") " pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.208003 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.207982 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb8b2\" (UniqueName: \"kubernetes.io/projected/75503659-d535-4d5b-9485-cd92d3a6cf09-kube-api-access-sb8b2\") pod \"openshift-state-metrics-67dcf8c979-h8wbz\" (UID: \"75503659-d535-4d5b-9485-cd92d3a6cf09\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.247992 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.247881 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-pkjx9"] Sep 11 14:16:56.250413 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:56.250394 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859033b0_6de0_484b_b73c_eaea2affa856.slice/crio-3f9232b6740798fe2771be079834204e5a1ca2690830b42e752bdb8430d3aba6 WatchSource:0}: Error finding container 3f9232b6740798fe2771be079834204e5a1ca2690830b42e752bdb8430d3aba6: Status 404 returned error can't find the container with id 3f9232b6740798fe2771be079834204e5a1ca2690830b42e752bdb8430d3aba6 Sep 11 14:16:56.320295 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.320272 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" Sep 11 14:16:56.347655 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.347637 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nz7g9" Sep 11 14:16:56.361490 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.361468 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" Sep 11 14:16:56.367639 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:56.367616 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558b6beb_1341_417b_9763_85053d4d8b26.slice/crio-bf801fdda2f4b2ec95205feadb4bbef75ea740aa38437641dfaff983e597dd93 WatchSource:0}: Error finding container bf801fdda2f4b2ec95205feadb4bbef75ea740aa38437641dfaff983e597dd93: Status 404 returned error can't find the container with id bf801fdda2f4b2ec95205feadb4bbef75ea740aa38437641dfaff983e597dd93 Sep 11 14:16:56.623948 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.623916 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz"] Sep 11 14:16:56.626697 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:56.626671 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75503659_d535_4d5b_9485_cd92d3a6cf09.slice/crio-32b1570ad97c624e3d1fb5b18d1de6b0dcf63fde9547075109975f7270fc41a6 WatchSource:0}: Error finding container 32b1570ad97c624e3d1fb5b18d1de6b0dcf63fde9547075109975f7270fc41a6: Status 404 returned error can't find the container with id 32b1570ad97c624e3d1fb5b18d1de6b0dcf63fde9547075109975f7270fc41a6 Sep 11 14:16:56.672850 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.672830 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s"] Sep 11 14:16:56.674690 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:16:56.674668 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4050a332_4649_4891_ac88_36a83d70b6ba.slice/crio-d78150523347642eeb4ac31d06284fedebb4d9a427936206a02482a68d5835dc WatchSource:0}: Error finding container d78150523347642eeb4ac31d06284fedebb4d9a427936206a02482a68d5835dc: Status 404 returned error can't find the container with id d78150523347642eeb4ac31d06284fedebb4d9a427936206a02482a68d5835dc Sep 11 14:16:56.899410 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.899333 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" event={"ID":"75503659-d535-4d5b-9485-cd92d3a6cf09","Type":"ContainerStarted","Data":"8da6616264cd5b9afbeea87dfeffd7e839470bd18a4bef6ebb118fa6747d7158"} Sep 11 14:16:56.899677 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.899652 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" event={"ID":"75503659-d535-4d5b-9485-cd92d3a6cf09","Type":"ContainerStarted","Data":"7effcd980c5c1f810ae96994ca06c23d7424dd519d1a27cbbca07e1db447ef56"} Sep 11 14:16:56.899842 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.899820 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" event={"ID":"75503659-d535-4d5b-9485-cd92d3a6cf09","Type":"ContainerStarted","Data":"32b1570ad97c624e3d1fb5b18d1de6b0dcf63fde9547075109975f7270fc41a6"} Sep 11 14:16:56.900696 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.900671 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nz7g9" event={"ID":"558b6beb-1341-417b-9763-85053d4d8b26","Type":"ContainerStarted","Data":"bf801fdda2f4b2ec95205feadb4bbef75ea740aa38437641dfaff983e597dd93"} Sep 11 14:16:56.901939 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.901914 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-pkjx9" event={"ID":"859033b0-6de0-484b-b73c-eaea2affa856","Type":"ContainerStarted","Data":"3f9232b6740798fe2771be079834204e5a1ca2690830b42e752bdb8430d3aba6"} Sep 11 14:16:56.903152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:56.903132 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" event={"ID":"4050a332-4649-4891-ac88-36a83d70b6ba","Type":"ContainerStarted","Data":"d78150523347642eeb4ac31d06284fedebb4d9a427936206a02482a68d5835dc"} Sep 11 14:16:57.124201 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.124161 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:16:57.147567 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.147241 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.150885 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.150388 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:16:57.150885 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.150677 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 14:16:57.150885 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.150685 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 14:16:57.151090 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.151026 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 14:16:57.151400 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.151273 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 14:16:57.151400 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.151286 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 14:16:57.152862 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.151826 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 14:16:57.152862 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.152021 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 14:16:57.152862 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.152251 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bktcp\"" Sep 11 14:16:57.152862 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.152449 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 14:16:57.152862 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.152612 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 14:16:57.201213 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.201187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-config-volume\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.201347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.201221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.201347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.201243 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.201347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.201267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.201347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.201337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.201546 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.201384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd730607-c83d-405c-8734-5ac80033df91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.201546 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.201426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.201546 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.201461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-web-config\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.201546 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.201491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.201546 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.201533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.201793 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.201593 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd730607-c83d-405c-8734-5ac80033df91-config-out\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.201793 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.201641 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.201793 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.201688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwtx5\" (UniqueName: \"kubernetes.io/projected/fd730607-c83d-405c-8734-5ac80033df91-kube-api-access-cwtx5\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.302394 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.302357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd730607-c83d-405c-8734-5ac80033df91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.302561 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.302401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.302561 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.302430 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-web-config\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.302561 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.302451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.302561 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.302491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.302561 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.302518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd730607-c83d-405c-8734-5ac80033df91-config-out\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.302561 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.302542 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.302894 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.302578 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwtx5\" (UniqueName: \"kubernetes.io/projected/fd730607-c83d-405c-8734-5ac80033df91-kube-api-access-cwtx5\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.302894 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.302627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-config-volume\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.302894 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.302655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.302894 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.302678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.302894 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.302700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.302894 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.302756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.305860 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:57.303754 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-trusted-ca-bundle podName:fd730607-c83d-405c-8734-5ac80033df91 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:57.803635672 +0000 UTC m=+105.889092025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "fd730607-c83d-405c-8734-5ac80033df91") : configmap references non-existent config key: ca-bundle.crt Sep 11 14:16:57.305860 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.303992 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.305860 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:57.304099 2578 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 11 14:16:57.305860 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:16:57.304148 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-main-tls podName:fd730607-c83d-405c-8734-5ac80033df91 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:57.804132285 +0000 UTC m=+105.889588627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "fd730607-c83d-405c-8734-5ac80033df91") : secret "alertmanager-main-tls" not found Sep 11 14:16:57.305860 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.305183 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd730607-c83d-405c-8734-5ac80033df91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.305860 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.305477 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.308703 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.306624 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.308703 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.307125 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-config-volume\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.310078 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.309327 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-web-config\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.310078 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.309406 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.310078 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.309808 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.310571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.310548 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd730607-c83d-405c-8734-5ac80033df91-config-out\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.310957 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.310930 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.319105 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.319081 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwtx5\" (UniqueName: \"kubernetes.io/projected/fd730607-c83d-405c-8734-5ac80033df91-kube-api-access-cwtx5\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.806283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.806073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.806283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.806123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.806906 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.806883 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.809711 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.809547 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:57.912796 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:57.912750 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nz7g9" event={"ID":"558b6beb-1341-417b-9763-85053d4d8b26","Type":"ContainerStarted","Data":"b963b70f4851d2f19e8ee953836aef112bd4c1aec15b777b77afe2922d0af29b"} Sep 11 14:16:58.062710 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:58.062673 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.919048 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:58.918913 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" event={"ID":"4050a332-4649-4891-ac88-36a83d70b6ba","Type":"ContainerStarted","Data":"0ba0811f6977a686834ab76b74c570dc6bfdd8512c8af13fcfe7204099fb58bb"} Sep 11 14:16:58.925115 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:58.925085 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" event={"ID":"75503659-d535-4d5b-9485-cd92d3a6cf09","Type":"ContainerStarted","Data":"a0d015b5ac5219dc58d6f4d772b66b166743559dc21223b442c65eb3c4751a55"} Sep 11 14:16:58.930156 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:58.929291 2578 generic.go:356] "Generic (PLEG): container finished" podID="558b6beb-1341-417b-9763-85053d4d8b26" containerID="b963b70f4851d2f19e8ee953836aef112bd4c1aec15b777b77afe2922d0af29b" exitCode=0 Sep 11 14:16:58.930156 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:58.929344 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nz7g9" event={"ID":"558b6beb-1341-417b-9763-85053d4d8b26","Type":"ContainerDied","Data":"b963b70f4851d2f19e8ee953836aef112bd4c1aec15b777b77afe2922d0af29b"} Sep 11 14:16:59.026127 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:59.026064 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-h8wbz" podStartSLOduration=2.05647613 podStartE2EDuration="4.026021389s" totalImagesPullingTime="1.969545259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:58.965062689 +0000 UTC m=+107.050519051" watchObservedRunningTime="2026-09-11 14:16:59.026021389 +0000 UTC m=+107.111477753" Sep 11 14:16:59.119998 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:59.119974 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:16:59.937027 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:59.936990 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" event={"ID":"4050a332-4649-4891-ac88-36a83d70b6ba","Type":"ContainerStarted","Data":"8f21d0ad1cd0f532fe57eaff7a492b2fa177be6c2a5a98bfb7335118bf8698f0"} Sep 11 14:16:59.937471 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:59.937057 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" event={"ID":"4050a332-4649-4891-ac88-36a83d70b6ba","Type":"ContainerStarted","Data":"8eafc9d5d8ce0e961a2687d7cbe11a6bc4677d08bb62c017681530bfb5d1dd19"} Sep 11 14:16:59.938472 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:59.938433 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerStarted","Data":"c8cda736f85f87c3915ebbd6c09f12674645cb3d92fcc0126b3cc657c6087cd1"} Sep 11 14:16:59.940845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:59.940789 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nz7g9" event={"ID":"558b6beb-1341-417b-9763-85053d4d8b26","Type":"ContainerStarted","Data":"1ad5fd1c8d9cafc5d650b630cc8186f01bd47332befdadda9804d001bab86b1b"} Sep 11 14:16:59.940845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:16:59.940838 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nz7g9" event={"ID":"558b6beb-1341-417b-9763-85053d4d8b26","Type":"ContainerStarted","Data":"7482e52c481458bcca82675fa92bcffc7f9f2305f13e52d31f47363f0154de10"} Sep 11 14:17:00.005687 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:00.005626 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lzz8s" podStartSLOduration=2.867940612 podStartE2EDuration="5.005607128s" totalImagesPullingTime="2.137666516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:59.967208844 +0000 UTC m=+108.052665206" watchObservedRunningTime="2026-09-11 14:17:00.005607128 +0000 UTC m=+108.091063490" Sep 11 14:17:00.006569 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:00.006521 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nz7g9" podStartSLOduration=3.533029364 podStartE2EDuration="5.006505823s" totalImagesPullingTime="1.365091162s" totalInitContainerRuntime="108.385297ms" isStatefulPod=true podCreationTimestamp="2026-09-11 14:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:00.005371527 +0000 UTC m=+108.090827889" watchObservedRunningTime="2026-09-11 14:17:00.006505823 +0000 UTC m=+108.091962182" Sep 11 14:17:00.946482 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:00.946402 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd730607-c83d-405c-8734-5ac80033df91" containerID="e5eff8451c811eafbf40b2a42a439bd289d9bc247219061d3de491d1f864f5f9" exitCode=0 Sep 11 14:17:00.946906 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:00.946500 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerDied","Data":"e5eff8451c811eafbf40b2a42a439bd289d9bc247219061d3de491d1f864f5f9"} Sep 11 14:17:02.960093 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:02.959994 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerStarted","Data":"e9e30a81fb57d2e6d879cf6bbfca10ac00abac43c3f4ab1b0ddcdf9b189a2d54"} Sep 11 14:17:02.960421 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:02.960115 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerStarted","Data":"6e54474bf2b7e7d9a0ea4ae00fee9d36681bebfad0c3ecf1ab7907e1dc4cbe46"} Sep 11 14:17:02.960421 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:02.960165 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerStarted","Data":"2d1274bd414f38ff5d7795ff549e4cdd0b663d2a1b1308fe2e98e8188bb9bcbd"} Sep 11 14:17:03.889258 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:03.889235 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qdqt7" Sep 11 14:17:03.970526 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:03.970488 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerStarted","Data":"0ab3f32e7e92f28caeec0deae42994519cf91ac07d09b3815c29c103d0b3cc74"} Sep 11 14:17:03.970981 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:03.970570 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerStarted","Data":"1a6a124df59ff24db6232f2060eacc1d1a649ec714a8fd26724c038272833348"} Sep 11 14:17:03.970981 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:03.970611 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerStarted","Data":"6cfd743f2d3f307aa6a27ca0b9530bef2cfe20052811b5ce5e5909a19d055d03"} Sep 11 14:17:04.032131 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:04.031945 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.035393901 podStartE2EDuration="7.031851851s" totalImagesPullingTime="3.973632309s" totalInitContainerRuntime="22.825641ms" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:04.023327355 +0000 UTC m=+112.108783718" watchObservedRunningTime="2026-09-11 14:17:04.031851851 +0000 UTC m=+112.117308212" Sep 11 14:17:14.014543 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:14.014499 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-pkjx9" event={"ID":"859033b0-6de0-484b-b73c-eaea2affa856","Type":"ContainerStarted","Data":"03170d51d6313f939626b7c5b647be5ff2cb4f26603cd376724a4f02493b6299"} Sep 11 14:17:14.015029 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:14.014762 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-pkjx9" Sep 11 14:17:14.016381 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:14.016354 2578 patch_prober.go:28] interesting pod/downloads-76f99584f4-pkjx9 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.16:8080/\": dial tcp 10.134.0.16:8080: connect: connection refused" start-of-body= Sep 11 14:17:14.016504 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:14.016420 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-pkjx9" podUID="859033b0-6de0-484b-b73c-eaea2affa856" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.16:8080/\": dial tcp 10.134.0.16:8080: connect: connection refused" Sep 11 14:17:14.065628 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:14.065567 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-pkjx9" podStartSLOduration=1.460228291 podStartE2EDuration="19.065546316s" totalImagesPullingTime="17.605318025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:14.062570319 +0000 UTC m=+122.148026679" watchObservedRunningTime="2026-09-11 14:17:14.065546316 +0000 UTC m=+122.151002678" Sep 11 14:17:15.026834 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:15.026796 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-pkjx9" Sep 11 14:17:16.737955 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.737922 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fcf9d69fd-4x7zt"] Sep 11 14:17:16.767006 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.766976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.770717 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.770688 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 14:17:16.771510 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.771486 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 14:17:16.772143 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.772085 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-llnzd\"" Sep 11 14:17:16.772246 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.772099 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 14:17:16.772591 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.772567 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 14:17:16.773668 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.773647 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 14:17:16.777385 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.777364 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 14:17:16.782754 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.782706 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fcf9d69fd-4x7zt"] Sep 11 14:17:16.881176 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.881140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-oauth-serving-cert\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.881375 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.881186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-serving-cert\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.881375 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.881232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-trusted-ca-bundle\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.881375 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.881256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlhpv\" (UniqueName: \"kubernetes.io/projected/d67db4d6-512f-4950-afeb-13e2b4a952a7-kube-api-access-zlhpv\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.881375 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.881341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-service-ca\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.881554 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.881384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-oauth-config\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.881554 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.881401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-config\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.982450 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.982414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-service-ca\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.982616 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.982463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-oauth-config\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.982616 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.982488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-config\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.982772 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.982675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-oauth-serving-cert\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.982772 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.982757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-serving-cert\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.982892 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.982798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-trusted-ca-bundle\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.982892 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.982822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlhpv\" (UniqueName: \"kubernetes.io/projected/d67db4d6-512f-4950-afeb-13e2b4a952a7-kube-api-access-zlhpv\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.983376 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.983344 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-service-ca\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.983489 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.983373 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-oauth-serving-cert\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.983808 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.983779 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-trusted-ca-bundle\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.983935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.983789 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-config\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.985787 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.985748 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-serving-cert\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.986023 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.985999 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-oauth-config\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:16.997590 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:16.997538 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlhpv\" (UniqueName: \"kubernetes.io/projected/d67db4d6-512f-4950-afeb-13e2b4a952a7-kube-api-access-zlhpv\") pod \"console-7fcf9d69fd-4x7zt\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:17.078811 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:17.078780 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:17.435295 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:17.431932 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fcf9d69fd-4x7zt"] Sep 11 14:17:17.439800 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:17:17.439754 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67db4d6_512f_4950_afeb_13e2b4a952a7.slice/crio-7062ee072493a7e0843e5bf5ab12122f563eaa4870235bb3a781845bb47ea435 WatchSource:0}: Error finding container 7062ee072493a7e0843e5bf5ab12122f563eaa4870235bb3a781845bb47ea435: Status 404 returned error can't find the container with id 7062ee072493a7e0843e5bf5ab12122f563eaa4870235bb3a781845bb47ea435 Sep 11 14:17:18.034794 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:18.034743 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fcf9d69fd-4x7zt" event={"ID":"d67db4d6-512f-4950-afeb-13e2b4a952a7","Type":"ContainerStarted","Data":"7062ee072493a7e0843e5bf5ab12122f563eaa4870235bb3a781845bb47ea435"} Sep 11 14:17:22.059287 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:22.059248 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fcf9d69fd-4x7zt" event={"ID":"d67db4d6-512f-4950-afeb-13e2b4a952a7","Type":"ContainerStarted","Data":"5cba4fd51d3b0d75347d677208895cdcbd79fb7c09b18b53c351e79eb19bfc6b"} Sep 11 14:17:22.087632 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:22.087588 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fcf9d69fd-4x7zt" podStartSLOduration=2.429077878 podStartE2EDuration="6.087569324s" totalImagesPullingTime="3.658491446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:22.085861502 +0000 UTC m=+130.171317863" watchObservedRunningTime="2026-09-11 14:17:22.087569324 +0000 UTC m=+130.173025683" Sep 11 14:17:22.131452 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:22.131421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:17:22.134123 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:22.134095 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df1160f9-2ec0-4907-9ca2-4b269875d943-metrics-certs\") pod \"network-metrics-daemon-pp2sq\" (UID: \"df1160f9-2ec0-4907-9ca2-4b269875d943\") " pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:17:22.179246 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:22.179221 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-96vl6\"" Sep 11 14:17:22.186848 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:22.186829 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pp2sq" Sep 11 14:17:22.487380 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:22.487299 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pp2sq"] Sep 11 14:17:22.490387 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:17:22.490357 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1160f9_2ec0_4907_9ca2_4b269875d943.slice/crio-0aa563bc16573b170b43652c3cb6e1152e20a92da400749d4e2c3a229a47b779 WatchSource:0}: Error finding container 0aa563bc16573b170b43652c3cb6e1152e20a92da400749d4e2c3a229a47b779: Status 404 returned error can't find the container with id 0aa563bc16573b170b43652c3cb6e1152e20a92da400749d4e2c3a229a47b779 Sep 11 14:17:23.064640 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:23.064594 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pp2sq" event={"ID":"df1160f9-2ec0-4907-9ca2-4b269875d943","Type":"ContainerStarted","Data":"0aa563bc16573b170b43652c3cb6e1152e20a92da400749d4e2c3a229a47b779"} Sep 11 14:17:25.074187 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:25.074151 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pp2sq" event={"ID":"df1160f9-2ec0-4907-9ca2-4b269875d943","Type":"ContainerStarted","Data":"a567e5b7d384b7e69a26daafd126dcd093bac3b83f7bb647988ae5b7c4e498e2"} Sep 11 14:17:25.074627 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:25.074206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pp2sq" event={"ID":"df1160f9-2ec0-4907-9ca2-4b269875d943","Type":"ContainerStarted","Data":"f08e9b71ee80a360579ac9ae48e514bdc9c0b352e2c82c5d80693bae3699145f"} Sep 11 14:17:25.101715 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:25.101671 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pp2sq" podStartSLOduration=131.59561022 podStartE2EDuration="2m13.101657001s" totalImagesPullingTime="1.506046781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:25.100852111 +0000 UTC m=+133.186308471" watchObservedRunningTime="2026-09-11 14:17:25.101657001 +0000 UTC m=+133.187113360" Sep 11 14:17:27.079195 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:27.079159 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:27.079654 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:27.079216 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:27.083932 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:27.083903 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:17:28.089196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:17:28.089167 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:18:16.857132 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:16.857097 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:18:16.857607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:16.857384 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="alertmanager" containerID="cri-o://2d1274bd414f38ff5d7795ff549e4cdd0b663d2a1b1308fe2e98e8188bb9bcbd" gracePeriod=120 Sep 11 14:18:16.857607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:16.857451 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="kube-rbac-proxy-web" containerID="cri-o://e9e30a81fb57d2e6d879cf6bbfca10ac00abac43c3f4ab1b0ddcdf9b189a2d54" gracePeriod=120 Sep 11 14:18:16.857607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:16.857494 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="prom-label-proxy" containerID="cri-o://0ab3f32e7e92f28caeec0deae42994519cf91ac07d09b3815c29c103d0b3cc74" gracePeriod=120 Sep 11 14:18:16.857607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:16.857507 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="kube-rbac-proxy" containerID="cri-o://6cfd743f2d3f307aa6a27ca0b9530bef2cfe20052811b5ce5e5909a19d055d03" gracePeriod=120 Sep 11 14:18:16.857607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:16.857472 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="config-reloader" containerID="cri-o://6e54474bf2b7e7d9a0ea4ae00fee9d36681bebfad0c3ecf1ab7907e1dc4cbe46" gracePeriod=120 Sep 11 14:18:16.857607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:16.857446 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="kube-rbac-proxy-metric" containerID="cri-o://1a6a124df59ff24db6232f2060eacc1d1a649ec714a8fd26724c038272833348" gracePeriod=120 Sep 11 14:18:17.272565 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:17.272489 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd730607-c83d-405c-8734-5ac80033df91" containerID="0ab3f32e7e92f28caeec0deae42994519cf91ac07d09b3815c29c103d0b3cc74" exitCode=0 Sep 11 14:18:17.272565 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:17.272514 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd730607-c83d-405c-8734-5ac80033df91" containerID="6cfd743f2d3f307aa6a27ca0b9530bef2cfe20052811b5ce5e5909a19d055d03" exitCode=0 Sep 11 14:18:17.272565 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:17.272521 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd730607-c83d-405c-8734-5ac80033df91" containerID="e9e30a81fb57d2e6d879cf6bbfca10ac00abac43c3f4ab1b0ddcdf9b189a2d54" exitCode=0 Sep 11 14:18:17.272565 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:17.272528 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd730607-c83d-405c-8734-5ac80033df91" containerID="6e54474bf2b7e7d9a0ea4ae00fee9d36681bebfad0c3ecf1ab7907e1dc4cbe46" exitCode=0 Sep 11 14:18:17.272565 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:17.272535 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd730607-c83d-405c-8734-5ac80033df91" containerID="2d1274bd414f38ff5d7795ff549e4cdd0b663d2a1b1308fe2e98e8188bb9bcbd" exitCode=0 Sep 11 14:18:17.272565 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:17.272557 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerDied","Data":"0ab3f32e7e92f28caeec0deae42994519cf91ac07d09b3815c29c103d0b3cc74"} Sep 11 14:18:17.272864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:17.272613 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerDied","Data":"6cfd743f2d3f307aa6a27ca0b9530bef2cfe20052811b5ce5e5909a19d055d03"} Sep 11 14:18:17.272864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:17.272638 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerDied","Data":"e9e30a81fb57d2e6d879cf6bbfca10ac00abac43c3f4ab1b0ddcdf9b189a2d54"} Sep 11 14:18:17.272864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:17.272661 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerDied","Data":"6e54474bf2b7e7d9a0ea4ae00fee9d36681bebfad0c3ecf1ab7907e1dc4cbe46"} Sep 11 14:18:17.272864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:17.272684 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerDied","Data":"2d1274bd414f38ff5d7795ff549e4cdd0b663d2a1b1308fe2e98e8188bb9bcbd"} Sep 11 14:18:18.098654 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.098631 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.164135 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.164102 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-trusted-ca-bundle\") pod \"fd730607-c83d-405c-8734-5ac80033df91\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " Sep 11 14:18:18.164298 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.164152 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd730607-c83d-405c-8734-5ac80033df91-config-out\") pod \"fd730607-c83d-405c-8734-5ac80033df91\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " Sep 11 14:18:18.164298 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.164198 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"fd730607-c83d-405c-8734-5ac80033df91\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " Sep 11 14:18:18.164298 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.164235 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd730607-c83d-405c-8734-5ac80033df91-tls-assets\") pod \"fd730607-c83d-405c-8734-5ac80033df91\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " Sep 11 14:18:18.164298 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.164261 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-web-config\") pod \"fd730607-c83d-405c-8734-5ac80033df91\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " Sep 11 14:18:18.164298 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.164290 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy\") pod \"fd730607-c83d-405c-8734-5ac80033df91\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " Sep 11 14:18:18.164571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.164322 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-cluster-tls-config\") pod \"fd730607-c83d-405c-8734-5ac80033df91\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " Sep 11 14:18:18.164571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.164351 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-metrics-client-ca\") pod \"fd730607-c83d-405c-8734-5ac80033df91\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " Sep 11 14:18:18.164571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.164384 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy-web\") pod \"fd730607-c83d-405c-8734-5ac80033df91\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " Sep 11 14:18:18.164571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.164422 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-main-tls\") pod \"fd730607-c83d-405c-8734-5ac80033df91\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " Sep 11 14:18:18.164571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.164460 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-main-db\") pod \"fd730607-c83d-405c-8734-5ac80033df91\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " Sep 11 14:18:18.164571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.164488 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-config-volume\") pod \"fd730607-c83d-405c-8734-5ac80033df91\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " Sep 11 14:18:18.164571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.164514 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwtx5\" (UniqueName: \"kubernetes.io/projected/fd730607-c83d-405c-8734-5ac80033df91-kube-api-access-cwtx5\") pod \"fd730607-c83d-405c-8734-5ac80033df91\" (UID: \"fd730607-c83d-405c-8734-5ac80033df91\") " Sep 11 14:18:18.165481 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.165152 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "fd730607-c83d-405c-8734-5ac80033df91" (UID: "fd730607-c83d-405c-8734-5ac80033df91"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:18.165948 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.165718 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "fd730607-c83d-405c-8734-5ac80033df91" (UID: "fd730607-c83d-405c-8734-5ac80033df91"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:18.165948 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.165843 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "fd730607-c83d-405c-8734-5ac80033df91" (UID: "fd730607-c83d-405c-8734-5ac80033df91"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:18:18.169137 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.169010 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "fd730607-c83d-405c-8734-5ac80033df91" (UID: "fd730607-c83d-405c-8734-5ac80033df91"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:18.169137 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.169064 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd730607-c83d-405c-8734-5ac80033df91-kube-api-access-cwtx5" (OuterVolumeSpecName: "kube-api-access-cwtx5") pod "fd730607-c83d-405c-8734-5ac80033df91" (UID: "fd730607-c83d-405c-8734-5ac80033df91"). InnerVolumeSpecName "kube-api-access-cwtx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:18:18.169137 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.169088 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd730607-c83d-405c-8734-5ac80033df91-config-out" (OuterVolumeSpecName: "config-out") pod "fd730607-c83d-405c-8734-5ac80033df91" (UID: "fd730607-c83d-405c-8734-5ac80033df91"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:18:18.169137 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.169088 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd730607-c83d-405c-8734-5ac80033df91-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fd730607-c83d-405c-8734-5ac80033df91" (UID: "fd730607-c83d-405c-8734-5ac80033df91"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:18:18.169382 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.169164 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-config-volume" (OuterVolumeSpecName: "config-volume") pod "fd730607-c83d-405c-8734-5ac80033df91" (UID: "fd730607-c83d-405c-8734-5ac80033df91"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:18.169847 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.169823 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "fd730607-c83d-405c-8734-5ac80033df91" (UID: "fd730607-c83d-405c-8734-5ac80033df91"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:18.170398 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.170374 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "fd730607-c83d-405c-8734-5ac80033df91" (UID: "fd730607-c83d-405c-8734-5ac80033df91"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:18.170571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.170544 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "fd730607-c83d-405c-8734-5ac80033df91" (UID: "fd730607-c83d-405c-8734-5ac80033df91"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:18.173628 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.173605 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "fd730607-c83d-405c-8734-5ac80033df91" (UID: "fd730607-c83d-405c-8734-5ac80033df91"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:18.180008 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.179985 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-web-config" (OuterVolumeSpecName: "web-config") pod "fd730607-c83d-405c-8734-5ac80033df91" (UID: "fd730607-c83d-405c-8734-5ac80033df91"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:18.265257 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.265169 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:18.265257 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.265201 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd730607-c83d-405c-8734-5ac80033df91-config-out\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:18.265257 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.265214 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:18.265257 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.265226 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd730607-c83d-405c-8734-5ac80033df91-tls-assets\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:18.265257 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.265238 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-web-config\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:18.265257 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.265249 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:18.265545 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.265261 2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-cluster-tls-config\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:18.265545 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.265273 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd730607-c83d-405c-8734-5ac80033df91-metrics-client-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:18.265545 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.265286 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:18.265545 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.265299 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-secret-alertmanager-main-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:18.265545 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.265312 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd730607-c83d-405c-8734-5ac80033df91-alertmanager-main-db\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:18.265545 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.265323 2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd730607-c83d-405c-8734-5ac80033df91-config-volume\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:18.265545 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.265334 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwtx5\" (UniqueName: \"kubernetes.io/projected/fd730607-c83d-405c-8734-5ac80033df91-kube-api-access-cwtx5\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:18.279265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.279235 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd730607-c83d-405c-8734-5ac80033df91" containerID="1a6a124df59ff24db6232f2060eacc1d1a649ec714a8fd26724c038272833348" exitCode=0 Sep 11 14:18:18.279382 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.279320 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.279424 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.279317 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerDied","Data":"1a6a124df59ff24db6232f2060eacc1d1a649ec714a8fd26724c038272833348"} Sep 11 14:18:18.279501 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.279485 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd730607-c83d-405c-8734-5ac80033df91","Type":"ContainerDied","Data":"c8cda736f85f87c3915ebbd6c09f12674645cb3d92fcc0126b3cc657c6087cd1"} Sep 11 14:18:18.279552 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.279541 2578 scope.go:117] "RemoveContainer" containerID="0ab3f32e7e92f28caeec0deae42994519cf91ac07d09b3815c29c103d0b3cc74" Sep 11 14:18:18.287352 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.287335 2578 scope.go:117] "RemoveContainer" containerID="1a6a124df59ff24db6232f2060eacc1d1a649ec714a8fd26724c038272833348" Sep 11 14:18:18.294353 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.294336 2578 scope.go:117] "RemoveContainer" containerID="6cfd743f2d3f307aa6a27ca0b9530bef2cfe20052811b5ce5e5909a19d055d03" Sep 11 14:18:18.300896 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.300880 2578 scope.go:117] "RemoveContainer" containerID="e9e30a81fb57d2e6d879cf6bbfca10ac00abac43c3f4ab1b0ddcdf9b189a2d54" Sep 11 14:18:18.307207 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.307192 2578 scope.go:117] "RemoveContainer" containerID="6e54474bf2b7e7d9a0ea4ae00fee9d36681bebfad0c3ecf1ab7907e1dc4cbe46" Sep 11 14:18:18.310070 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.310048 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:18:18.314146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.314117 2578 scope.go:117] "RemoveContainer" containerID="2d1274bd414f38ff5d7795ff549e4cdd0b663d2a1b1308fe2e98e8188bb9bcbd" Sep 11 14:18:18.320079 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.320059 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:18:18.321354 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.321328 2578 scope.go:117] "RemoveContainer" containerID="e5eff8451c811eafbf40b2a42a439bd289d9bc247219061d3de491d1f864f5f9" Sep 11 14:18:18.327827 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.327812 2578 scope.go:117] "RemoveContainer" containerID="0ab3f32e7e92f28caeec0deae42994519cf91ac07d09b3815c29c103d0b3cc74" Sep 11 14:18:18.328108 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:18:18.328087 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab3f32e7e92f28caeec0deae42994519cf91ac07d09b3815c29c103d0b3cc74\": container with ID starting with 0ab3f32e7e92f28caeec0deae42994519cf91ac07d09b3815c29c103d0b3cc74 not found: ID does not exist" containerID="0ab3f32e7e92f28caeec0deae42994519cf91ac07d09b3815c29c103d0b3cc74" Sep 11 14:18:18.328167 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.328115 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab3f32e7e92f28caeec0deae42994519cf91ac07d09b3815c29c103d0b3cc74"} err="failed to get container status \"0ab3f32e7e92f28caeec0deae42994519cf91ac07d09b3815c29c103d0b3cc74\": rpc error: code = NotFound desc = could not find container \"0ab3f32e7e92f28caeec0deae42994519cf91ac07d09b3815c29c103d0b3cc74\": container with ID starting with 0ab3f32e7e92f28caeec0deae42994519cf91ac07d09b3815c29c103d0b3cc74 not found: ID does not exist" Sep 11 14:18:18.328167 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.328145 2578 scope.go:117] "RemoveContainer" containerID="1a6a124df59ff24db6232f2060eacc1d1a649ec714a8fd26724c038272833348" Sep 11 14:18:18.328398 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:18:18.328381 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a6a124df59ff24db6232f2060eacc1d1a649ec714a8fd26724c038272833348\": container with ID starting with 1a6a124df59ff24db6232f2060eacc1d1a649ec714a8fd26724c038272833348 not found: ID does not exist" containerID="1a6a124df59ff24db6232f2060eacc1d1a649ec714a8fd26724c038272833348" Sep 11 14:18:18.328437 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.328404 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a6a124df59ff24db6232f2060eacc1d1a649ec714a8fd26724c038272833348"} err="failed to get container status \"1a6a124df59ff24db6232f2060eacc1d1a649ec714a8fd26724c038272833348\": rpc error: code = NotFound desc = could not find container \"1a6a124df59ff24db6232f2060eacc1d1a649ec714a8fd26724c038272833348\": container with ID starting with 1a6a124df59ff24db6232f2060eacc1d1a649ec714a8fd26724c038272833348 not found: ID does not exist" Sep 11 14:18:18.328437 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.328420 2578 scope.go:117] "RemoveContainer" containerID="6cfd743f2d3f307aa6a27ca0b9530bef2cfe20052811b5ce5e5909a19d055d03" Sep 11 14:18:18.328642 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:18:18.328627 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cfd743f2d3f307aa6a27ca0b9530bef2cfe20052811b5ce5e5909a19d055d03\": container with ID starting with 6cfd743f2d3f307aa6a27ca0b9530bef2cfe20052811b5ce5e5909a19d055d03 not found: ID does not exist" containerID="6cfd743f2d3f307aa6a27ca0b9530bef2cfe20052811b5ce5e5909a19d055d03" Sep 11 14:18:18.328694 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.328645 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cfd743f2d3f307aa6a27ca0b9530bef2cfe20052811b5ce5e5909a19d055d03"} err="failed to get container status \"6cfd743f2d3f307aa6a27ca0b9530bef2cfe20052811b5ce5e5909a19d055d03\": rpc error: code = NotFound desc = could not find container \"6cfd743f2d3f307aa6a27ca0b9530bef2cfe20052811b5ce5e5909a19d055d03\": container with ID starting with 6cfd743f2d3f307aa6a27ca0b9530bef2cfe20052811b5ce5e5909a19d055d03 not found: ID does not exist" Sep 11 14:18:18.328694 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.328657 2578 scope.go:117] "RemoveContainer" containerID="e9e30a81fb57d2e6d879cf6bbfca10ac00abac43c3f4ab1b0ddcdf9b189a2d54" Sep 11 14:18:18.328876 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:18:18.328860 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e30a81fb57d2e6d879cf6bbfca10ac00abac43c3f4ab1b0ddcdf9b189a2d54\": container with ID starting with e9e30a81fb57d2e6d879cf6bbfca10ac00abac43c3f4ab1b0ddcdf9b189a2d54 not found: ID does not exist" containerID="e9e30a81fb57d2e6d879cf6bbfca10ac00abac43c3f4ab1b0ddcdf9b189a2d54" Sep 11 14:18:18.328923 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.328879 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e30a81fb57d2e6d879cf6bbfca10ac00abac43c3f4ab1b0ddcdf9b189a2d54"} err="failed to get container status \"e9e30a81fb57d2e6d879cf6bbfca10ac00abac43c3f4ab1b0ddcdf9b189a2d54\": rpc error: code = NotFound desc = could not find container \"e9e30a81fb57d2e6d879cf6bbfca10ac00abac43c3f4ab1b0ddcdf9b189a2d54\": container with ID starting with e9e30a81fb57d2e6d879cf6bbfca10ac00abac43c3f4ab1b0ddcdf9b189a2d54 not found: ID does not exist" Sep 11 14:18:18.328923 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.328891 2578 scope.go:117] "RemoveContainer" containerID="6e54474bf2b7e7d9a0ea4ae00fee9d36681bebfad0c3ecf1ab7907e1dc4cbe46" Sep 11 14:18:18.329095 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:18:18.329080 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e54474bf2b7e7d9a0ea4ae00fee9d36681bebfad0c3ecf1ab7907e1dc4cbe46\": container with ID starting with 6e54474bf2b7e7d9a0ea4ae00fee9d36681bebfad0c3ecf1ab7907e1dc4cbe46 not found: ID does not exist" containerID="6e54474bf2b7e7d9a0ea4ae00fee9d36681bebfad0c3ecf1ab7907e1dc4cbe46" Sep 11 14:18:18.329127 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.329101 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e54474bf2b7e7d9a0ea4ae00fee9d36681bebfad0c3ecf1ab7907e1dc4cbe46"} err="failed to get container status \"6e54474bf2b7e7d9a0ea4ae00fee9d36681bebfad0c3ecf1ab7907e1dc4cbe46\": rpc error: code = NotFound desc = could not find container \"6e54474bf2b7e7d9a0ea4ae00fee9d36681bebfad0c3ecf1ab7907e1dc4cbe46\": container with ID starting with 6e54474bf2b7e7d9a0ea4ae00fee9d36681bebfad0c3ecf1ab7907e1dc4cbe46 not found: ID does not exist" Sep 11 14:18:18.329127 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.329112 2578 scope.go:117] "RemoveContainer" containerID="2d1274bd414f38ff5d7795ff549e4cdd0b663d2a1b1308fe2e98e8188bb9bcbd" Sep 11 14:18:18.329344 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:18:18.329330 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1274bd414f38ff5d7795ff549e4cdd0b663d2a1b1308fe2e98e8188bb9bcbd\": container with ID starting with 2d1274bd414f38ff5d7795ff549e4cdd0b663d2a1b1308fe2e98e8188bb9bcbd not found: ID does not exist" containerID="2d1274bd414f38ff5d7795ff549e4cdd0b663d2a1b1308fe2e98e8188bb9bcbd" Sep 11 14:18:18.329377 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.329348 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1274bd414f38ff5d7795ff549e4cdd0b663d2a1b1308fe2e98e8188bb9bcbd"} err="failed to get container status \"2d1274bd414f38ff5d7795ff549e4cdd0b663d2a1b1308fe2e98e8188bb9bcbd\": rpc error: code = NotFound desc = could not find container \"2d1274bd414f38ff5d7795ff549e4cdd0b663d2a1b1308fe2e98e8188bb9bcbd\": container with ID starting with 2d1274bd414f38ff5d7795ff549e4cdd0b663d2a1b1308fe2e98e8188bb9bcbd not found: ID does not exist" Sep 11 14:18:18.329377 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.329370 2578 scope.go:117] "RemoveContainer" containerID="e5eff8451c811eafbf40b2a42a439bd289d9bc247219061d3de491d1f864f5f9" Sep 11 14:18:18.329591 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:18:18.329573 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5eff8451c811eafbf40b2a42a439bd289d9bc247219061d3de491d1f864f5f9\": container with ID starting with e5eff8451c811eafbf40b2a42a439bd289d9bc247219061d3de491d1f864f5f9 not found: ID does not exist" containerID="e5eff8451c811eafbf40b2a42a439bd289d9bc247219061d3de491d1f864f5f9" Sep 11 14:18:18.329591 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.329595 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5eff8451c811eafbf40b2a42a439bd289d9bc247219061d3de491d1f864f5f9"} err="failed to get container status \"e5eff8451c811eafbf40b2a42a439bd289d9bc247219061d3de491d1f864f5f9\": rpc error: code = NotFound desc = could not find container \"e5eff8451c811eafbf40b2a42a439bd289d9bc247219061d3de491d1f864f5f9\": container with ID starting with e5eff8451c811eafbf40b2a42a439bd289d9bc247219061d3de491d1f864f5f9 not found: ID does not exist" Sep 11 14:18:18.362617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362588 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:18:18.362887 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362873 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="init-config-reloader" Sep 11 14:18:18.362935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362888 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="init-config-reloader" Sep 11 14:18:18.362935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362897 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="kube-rbac-proxy-metric" Sep 11 14:18:18.362935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362902 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="kube-rbac-proxy-metric" Sep 11 14:18:18.362935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362909 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="kube-rbac-proxy-web" Sep 11 14:18:18.362935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362916 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="kube-rbac-proxy-web" Sep 11 14:18:18.362935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362924 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="kube-rbac-proxy" Sep 11 14:18:18.362935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362929 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="kube-rbac-proxy" Sep 11 14:18:18.363146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362936 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="prom-label-proxy" Sep 11 14:18:18.363146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362957 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="prom-label-proxy" Sep 11 14:18:18.363146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362965 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="config-reloader" Sep 11 14:18:18.363146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362970 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="config-reloader" Sep 11 14:18:18.363146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362978 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="alertmanager" Sep 11 14:18:18.363146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.362983 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="alertmanager" Sep 11 14:18:18.363146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.363028 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="alertmanager" Sep 11 14:18:18.363146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.363034 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="kube-rbac-proxy-metric" Sep 11 14:18:18.363146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.363041 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="kube-rbac-proxy-web" Sep 11 14:18:18.363146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.363047 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="config-reloader" Sep 11 14:18:18.363146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.363052 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="kube-rbac-proxy" Sep 11 14:18:18.363146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.363057 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd730607-c83d-405c-8734-5ac80033df91" containerName="prom-label-proxy" Sep 11 14:18:18.368244 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.368227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.371366 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.371345 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 14:18:18.371482 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.371385 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 14:18:18.371482 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.371427 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 14:18:18.371482 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.371464 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 14:18:18.371482 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.371473 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 14:18:18.371770 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.371717 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bktcp\"" Sep 11 14:18:18.371770 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.371761 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 14:18:18.372207 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.371897 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 14:18:18.372207 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.372119 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 14:18:18.377264 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.377248 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 14:18:18.392626 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.392605 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:18:18.466352 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466323 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd730607-c83d-405c-8734-5ac80033df91" path="/var/lib/kubelet/pods/fd730607-c83d-405c-8734-5ac80033df91/volumes" Sep 11 14:18:18.466568 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-web-config\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.466607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466579 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.466607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.466694 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466675 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c6b33d1-1111-48ec-8b02-11d055ac3e13-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.466745 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466711 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c6b33d1-1111-48ec-8b02-11d055ac3e13-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.466786 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466746 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.466786 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466768 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.466849 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c6b33d1-1111-48ec-8b02-11d055ac3e13-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.466849 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lpzc\" (UniqueName: \"kubernetes.io/projected/4c6b33d1-1111-48ec-8b02-11d055ac3e13-kube-api-access-2lpzc\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.466909 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c6b33d1-1111-48ec-8b02-11d055ac3e13-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.466909 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.467014 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c6b33d1-1111-48ec-8b02-11d055ac3e13-config-out\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.467014 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.466947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.568254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.568177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c6b33d1-1111-48ec-8b02-11d055ac3e13-config-out\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.568254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.568213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.568254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.568234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-web-config\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.568491 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.568257 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.568491 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.568281 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.568491 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.568315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c6b33d1-1111-48ec-8b02-11d055ac3e13-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.568491 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.568342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c6b33d1-1111-48ec-8b02-11d055ac3e13-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.568491 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.568369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.568491 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.568399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.568491 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.568439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c6b33d1-1111-48ec-8b02-11d055ac3e13-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.568491 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.568463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lpzc\" (UniqueName: \"kubernetes.io/projected/4c6b33d1-1111-48ec-8b02-11d055ac3e13-kube-api-access-2lpzc\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.568491 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.568493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c6b33d1-1111-48ec-8b02-11d055ac3e13-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.568951 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.568518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.569260 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.569226 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c6b33d1-1111-48ec-8b02-11d055ac3e13-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.569415 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.569394 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c6b33d1-1111-48ec-8b02-11d055ac3e13-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.569535 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.569511 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c6b33d1-1111-48ec-8b02-11d055ac3e13-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.571496 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.571161 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c6b33d1-1111-48ec-8b02-11d055ac3e13-config-out\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.571496 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.571306 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.571496 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.571325 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.571496 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.571431 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.571496 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.571438 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.571776 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.571674 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-web-config\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.571970 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.571948 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.572004 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.571969 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c6b33d1-1111-48ec-8b02-11d055ac3e13-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.573160 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.573144 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c6b33d1-1111-48ec-8b02-11d055ac3e13-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.584880 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.584863 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lpzc\" (UniqueName: \"kubernetes.io/projected/4c6b33d1-1111-48ec-8b02-11d055ac3e13-kube-api-access-2lpzc\") pod \"alertmanager-main-0\" (UID: \"4c6b33d1-1111-48ec-8b02-11d055ac3e13\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.677407 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.677385 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:18:18.965917 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:18.965885 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:18:18.969134 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:18:18.969103 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6b33d1_1111_48ec_8b02_11d055ac3e13.slice/crio-69f78412cdc8b33f383462b485c890e91dede3877e3c96d1b033df1b6a760d57 WatchSource:0}: Error finding container 69f78412cdc8b33f383462b485c890e91dede3877e3c96d1b033df1b6a760d57: Status 404 returned error can't find the container with id 69f78412cdc8b33f383462b485c890e91dede3877e3c96d1b033df1b6a760d57 Sep 11 14:18:19.285523 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:19.285434 2578 generic.go:356] "Generic (PLEG): container finished" podID="4c6b33d1-1111-48ec-8b02-11d055ac3e13" containerID="a2b3f6d8d0bd32c35468e0b53c2ba6cfef63c3d9c850e3aa297f8659e99ae103" exitCode=0 Sep 11 14:18:19.285523 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:19.285498 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c6b33d1-1111-48ec-8b02-11d055ac3e13","Type":"ContainerDied","Data":"a2b3f6d8d0bd32c35468e0b53c2ba6cfef63c3d9c850e3aa297f8659e99ae103"} Sep 11 14:18:19.286026 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:19.285568 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c6b33d1-1111-48ec-8b02-11d055ac3e13","Type":"ContainerStarted","Data":"69f78412cdc8b33f383462b485c890e91dede3877e3c96d1b033df1b6a760d57"} Sep 11 14:18:20.293831 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.293754 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c6b33d1-1111-48ec-8b02-11d055ac3e13","Type":"ContainerStarted","Data":"8d668a9aaab95f1dfe9b5f306f9cdce341015f3ebdbef18ed64554afd0122052"} Sep 11 14:18:20.293831 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.293818 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c6b33d1-1111-48ec-8b02-11d055ac3e13","Type":"ContainerStarted","Data":"b5d70363d82e604b0277046d5be5b5a6b7c8351e638f6ef787bd446b32dda762"} Sep 11 14:18:20.294183 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.293849 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c6b33d1-1111-48ec-8b02-11d055ac3e13","Type":"ContainerStarted","Data":"8c422f14d45115b24111417d1ce5e5b32e27bca31d67af0c86cf1c338888051a"} Sep 11 14:18:20.294183 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.293874 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c6b33d1-1111-48ec-8b02-11d055ac3e13","Type":"ContainerStarted","Data":"134fdf1baaf7c088467ad40f399e3b71611e4f3a3f748746cf7cbc543ead864a"} Sep 11 14:18:20.294183 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.293897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c6b33d1-1111-48ec-8b02-11d055ac3e13","Type":"ContainerStarted","Data":"03567c3e92648cc940d67cb67f4b21cce7c2ab6af97d4a6ce32e813880cddc7c"} Sep 11 14:18:20.294183 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.293925 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c6b33d1-1111-48ec-8b02-11d055ac3e13","Type":"ContainerStarted","Data":"d275a4f5f80d46ec45ea034dba9107ffc5861adf82b3a0ddcb45f033b7d8bd33"} Sep 11 14:18:20.332898 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.332855 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.317280376 podStartE2EDuration="2.332836983s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.556607ms" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:20.331809774 +0000 UTC m=+188.417266134" watchObservedRunningTime="2026-09-11 14:18:20.332836983 +0000 UTC m=+188.418293344" Sep 11 14:18:20.946674 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.946634 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd"] Sep 11 14:18:20.949302 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.949281 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:20.953631 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.953607 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 11 14:18:20.961041 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.961010 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 11 14:18:20.961126 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.961060 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 11 14:18:20.961296 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.961280 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 11 14:18:20.961335 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.961308 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6qw67\"" Sep 11 14:18:20.962621 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.962605 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 11 14:18:20.970319 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.970298 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 11 14:18:20.984885 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:20.984863 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd"] Sep 11 14:18:21.087678 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.087642 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aede382b-4542-4dfc-843e-f142f3115661-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.087678 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.087685 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aede382b-4542-4dfc-843e-f142f3115661-telemeter-client-tls\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.087939 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.087709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aede382b-4542-4dfc-843e-f142f3115661-federate-client-tls\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.087939 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.087791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aede382b-4542-4dfc-843e-f142f3115661-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.087939 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.087859 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aede382b-4542-4dfc-843e-f142f3115661-metrics-client-ca\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.087939 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.087911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aede382b-4542-4dfc-843e-f142f3115661-secret-telemeter-client\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.087939 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.087938 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzllz\" (UniqueName: \"kubernetes.io/projected/aede382b-4542-4dfc-843e-f142f3115661-kube-api-access-vzllz\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.088191 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.087962 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aede382b-4542-4dfc-843e-f142f3115661-serving-certs-ca-bundle\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.189046 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.189012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aede382b-4542-4dfc-843e-f142f3115661-telemeter-client-tls\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.189046 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.189061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aede382b-4542-4dfc-843e-f142f3115661-federate-client-tls\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.189260 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.189185 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aede382b-4542-4dfc-843e-f142f3115661-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.189260 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.189237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aede382b-4542-4dfc-843e-f142f3115661-metrics-client-ca\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.189377 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.189280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aede382b-4542-4dfc-843e-f142f3115661-secret-telemeter-client\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.189377 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.189322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzllz\" (UniqueName: \"kubernetes.io/projected/aede382b-4542-4dfc-843e-f142f3115661-kube-api-access-vzllz\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.189377 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.189342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aede382b-4542-4dfc-843e-f142f3115661-serving-certs-ca-bundle\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.189523 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.189387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aede382b-4542-4dfc-843e-f142f3115661-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.190019 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.189994 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aede382b-4542-4dfc-843e-f142f3115661-metrics-client-ca\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.190287 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.190256 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aede382b-4542-4dfc-843e-f142f3115661-serving-certs-ca-bundle\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.190417 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.190317 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aede382b-4542-4dfc-843e-f142f3115661-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.191796 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.191767 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aede382b-4542-4dfc-843e-f142f3115661-secret-telemeter-client\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.191894 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.191869 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aede382b-4542-4dfc-843e-f142f3115661-federate-client-tls\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.192075 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.192053 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aede382b-4542-4dfc-843e-f142f3115661-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.192255 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.192232 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aede382b-4542-4dfc-843e-f142f3115661-telemeter-client-tls\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.203651 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.203596 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzllz\" (UniqueName: \"kubernetes.io/projected/aede382b-4542-4dfc-843e-f142f3115661-kube-api-access-vzllz\") pod \"telemeter-client-c7bbdf7f8-wtvwd\" (UID: \"aede382b-4542-4dfc-843e-f142f3115661\") " pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.259471 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.259439 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" Sep 11 14:18:21.565552 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:21.565465 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd"] Sep 11 14:18:21.568588 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:18:21.568556 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaede382b_4542_4dfc_843e_f142f3115661.slice/crio-e4dab7c4d5e8ca32220b9b56839ecf69ada13de28a0653a314b38ebdce8a6364 WatchSource:0}: Error finding container e4dab7c4d5e8ca32220b9b56839ecf69ada13de28a0653a314b38ebdce8a6364: Status 404 returned error can't find the container with id e4dab7c4d5e8ca32220b9b56839ecf69ada13de28a0653a314b38ebdce8a6364 Sep 11 14:18:22.305472 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:22.305430 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" event={"ID":"aede382b-4542-4dfc-843e-f142f3115661","Type":"ContainerStarted","Data":"e4dab7c4d5e8ca32220b9b56839ecf69ada13de28a0653a314b38ebdce8a6364"} Sep 11 14:18:23.311259 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:23.311224 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" event={"ID":"aede382b-4542-4dfc-843e-f142f3115661","Type":"ContainerStarted","Data":"1d19a75688a1e0c2c6aceeb218da7929274165e47c69d2a51ba20801a7a29ff1"} Sep 11 14:18:23.311620 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:23.311312 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" event={"ID":"aede382b-4542-4dfc-843e-f142f3115661","Type":"ContainerStarted","Data":"06a7cb8aed5b5e2de3fd9b4ae68c587733f1cb0c4782e9729284512bb20e1ea6"} Sep 11 14:18:23.311620 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:23.311331 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" event={"ID":"aede382b-4542-4dfc-843e-f142f3115661","Type":"ContainerStarted","Data":"eb9a3516cc18e09ba88911f667e7451d775b661dedcdeec156f837a2e68238fe"} Sep 11 14:18:23.350635 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:23.350586 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-c7bbdf7f8-wtvwd" podStartSLOduration=1.836433861 podStartE2EDuration="3.35057089s" totalImagesPullingTime="1.514137029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:23.348508428 +0000 UTC m=+191.433964789" watchObservedRunningTime="2026-09-11 14:18:23.35057089 +0000 UTC m=+191.436027250" Sep 11 14:18:34.410552 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:34.410520 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fcf9d69fd-4x7zt"] Sep 11 14:18:59.432659 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.432590 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fcf9d69fd-4x7zt" podUID="d67db4d6-512f-4950-afeb-13e2b4a952a7" containerName="console" containerID="cri-o://5cba4fd51d3b0d75347d677208895cdcbd79fb7c09b18b53c351e79eb19bfc6b" gracePeriod=15 Sep 11 14:18:59.676754 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.676716 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:18:59.693305 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.693265 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-config\") pod \"d67db4d6-512f-4950-afeb-13e2b4a952a7\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " Sep 11 14:18:59.693305 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.693295 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-serving-cert\") pod \"d67db4d6-512f-4950-afeb-13e2b4a952a7\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " Sep 11 14:18:59.693402 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.693315 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-trusted-ca-bundle\") pod \"d67db4d6-512f-4950-afeb-13e2b4a952a7\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " Sep 11 14:18:59.693402 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.693335 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-service-ca\") pod \"d67db4d6-512f-4950-afeb-13e2b4a952a7\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " Sep 11 14:18:59.693402 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.693369 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-oauth-serving-cert\") pod \"d67db4d6-512f-4950-afeb-13e2b4a952a7\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " Sep 11 14:18:59.693530 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.693411 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlhpv\" (UniqueName: \"kubernetes.io/projected/d67db4d6-512f-4950-afeb-13e2b4a952a7-kube-api-access-zlhpv\") pod \"d67db4d6-512f-4950-afeb-13e2b4a952a7\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " Sep 11 14:18:59.693530 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.693453 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-oauth-config\") pod \"d67db4d6-512f-4950-afeb-13e2b4a952a7\" (UID: \"d67db4d6-512f-4950-afeb-13e2b4a952a7\") " Sep 11 14:18:59.693828 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.693795 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-service-ca" (OuterVolumeSpecName: "service-ca") pod "d67db4d6-512f-4950-afeb-13e2b4a952a7" (UID: "d67db4d6-512f-4950-afeb-13e2b4a952a7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:59.693921 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.693820 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d67db4d6-512f-4950-afeb-13e2b4a952a7" (UID: "d67db4d6-512f-4950-afeb-13e2b4a952a7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:59.693921 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.693846 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-config" (OuterVolumeSpecName: "console-config") pod "d67db4d6-512f-4950-afeb-13e2b4a952a7" (UID: "d67db4d6-512f-4950-afeb-13e2b4a952a7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:59.693921 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.693889 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d67db4d6-512f-4950-afeb-13e2b4a952a7" (UID: "d67db4d6-512f-4950-afeb-13e2b4a952a7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:59.695405 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.695377 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d67db4d6-512f-4950-afeb-13e2b4a952a7" (UID: "d67db4d6-512f-4950-afeb-13e2b4a952a7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:59.695511 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.695491 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d67db4d6-512f-4950-afeb-13e2b4a952a7" (UID: "d67db4d6-512f-4950-afeb-13e2b4a952a7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:18:59.695555 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.695525 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67db4d6-512f-4950-afeb-13e2b4a952a7-kube-api-access-zlhpv" (OuterVolumeSpecName: "kube-api-access-zlhpv") pod "d67db4d6-512f-4950-afeb-13e2b4a952a7" (UID: "d67db4d6-512f-4950-afeb-13e2b4a952a7"). InnerVolumeSpecName "kube-api-access-zlhpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:18:59.794912 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.794886 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-oauth-config\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:59.794912 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.794909 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-config\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:59.794912 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.794919 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d67db4d6-512f-4950-afeb-13e2b4a952a7-console-serving-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:59.795104 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.794928 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-trusted-ca-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:59.795104 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.794936 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:59.795104 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.794945 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d67db4d6-512f-4950-afeb-13e2b4a952a7-oauth-serving-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:18:59.795104 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:18:59.794952 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlhpv\" (UniqueName: \"kubernetes.io/projected/d67db4d6-512f-4950-afeb-13e2b4a952a7-kube-api-access-zlhpv\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:19:00.456607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:19:00.456570 2578 generic.go:356] "Generic (PLEG): container finished" podID="d67db4d6-512f-4950-afeb-13e2b4a952a7" containerID="5cba4fd51d3b0d75347d677208895cdcbd79fb7c09b18b53c351e79eb19bfc6b" exitCode=0 Sep 11 14:19:00.457011 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:19:00.456620 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fcf9d69fd-4x7zt" Sep 11 14:19:00.457011 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:19:00.456652 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fcf9d69fd-4x7zt" event={"ID":"d67db4d6-512f-4950-afeb-13e2b4a952a7","Type":"ContainerDied","Data":"5cba4fd51d3b0d75347d677208895cdcbd79fb7c09b18b53c351e79eb19bfc6b"} Sep 11 14:19:00.457011 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:19:00.456701 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fcf9d69fd-4x7zt" event={"ID":"d67db4d6-512f-4950-afeb-13e2b4a952a7","Type":"ContainerDied","Data":"7062ee072493a7e0843e5bf5ab12122f563eaa4870235bb3a781845bb47ea435"} Sep 11 14:19:00.457011 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:19:00.456720 2578 scope.go:117] "RemoveContainer" containerID="5cba4fd51d3b0d75347d677208895cdcbd79fb7c09b18b53c351e79eb19bfc6b" Sep 11 14:19:00.465409 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:19:00.465391 2578 scope.go:117] "RemoveContainer" containerID="5cba4fd51d3b0d75347d677208895cdcbd79fb7c09b18b53c351e79eb19bfc6b" Sep 11 14:19:00.465676 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:19:00.465656 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cba4fd51d3b0d75347d677208895cdcbd79fb7c09b18b53c351e79eb19bfc6b\": container with ID starting with 5cba4fd51d3b0d75347d677208895cdcbd79fb7c09b18b53c351e79eb19bfc6b not found: ID does not exist" containerID="5cba4fd51d3b0d75347d677208895cdcbd79fb7c09b18b53c351e79eb19bfc6b" Sep 11 14:19:00.465765 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:19:00.465690 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cba4fd51d3b0d75347d677208895cdcbd79fb7c09b18b53c351e79eb19bfc6b"} err="failed to get container status \"5cba4fd51d3b0d75347d677208895cdcbd79fb7c09b18b53c351e79eb19bfc6b\": rpc error: code = NotFound desc = could not find container \"5cba4fd51d3b0d75347d677208895cdcbd79fb7c09b18b53c351e79eb19bfc6b\": container with ID starting with 5cba4fd51d3b0d75347d677208895cdcbd79fb7c09b18b53c351e79eb19bfc6b not found: ID does not exist" Sep 11 14:19:02.751399 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:19:02.751365 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fcf9d69fd-4x7zt"] Sep 11 14:19:02.767089 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:19:02.767064 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fcf9d69fd-4x7zt"] Sep 11 14:19:04.463428 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:19:04.463386 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d67db4d6-512f-4950-afeb-13e2b4a952a7" path="/var/lib/kubelet/pods/d67db4d6-512f-4950-afeb-13e2b4a952a7/volumes" Sep 11 14:20:12.228720 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:12.228689 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:20:12.232877 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:12.232858 2578 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 14:20:12.234183 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:12.234167 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:20:32.031000 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.030967 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l"] Sep 11 14:20:32.033663 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.031259 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d67db4d6-512f-4950-afeb-13e2b4a952a7" containerName="console" Sep 11 14:20:32.033663 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.031269 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d67db4d6-512f-4950-afeb-13e2b4a952a7" containerName="console" Sep 11 14:20:32.033663 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.031335 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d67db4d6-512f-4950-afeb-13e2b4a952a7" containerName="console" Sep 11 14:20:32.034544 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.034528 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" Sep 11 14:20:32.037752 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.037719 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8txz9\"" Sep 11 14:20:32.040717 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.040663 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:20:32.041220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.041198 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:20:32.056602 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.056579 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l"] Sep 11 14:20:32.091605 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.091576 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l\" (UID: \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" Sep 11 14:20:32.091712 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.091623 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l\" (UID: \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" Sep 11 14:20:32.091712 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.091649 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdw7m\" (UniqueName: \"kubernetes.io/projected/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-kube-api-access-qdw7m\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l\" (UID: \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" Sep 11 14:20:32.192210 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.192182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdw7m\" (UniqueName: \"kubernetes.io/projected/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-kube-api-access-qdw7m\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l\" (UID: \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" Sep 11 14:20:32.192330 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.192236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l\" (UID: \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" Sep 11 14:20:32.192330 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.192276 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l\" (UID: \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" Sep 11 14:20:32.192679 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.192658 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l\" (UID: \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" Sep 11 14:20:32.192716 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.192674 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l\" (UID: \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" Sep 11 14:20:32.203664 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.203636 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdw7m\" (UniqueName: \"kubernetes.io/projected/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-kube-api-access-qdw7m\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l\" (UID: \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" Sep 11 14:20:32.342783 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.342693 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" Sep 11 14:20:32.625519 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.625411 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l"] Sep 11 14:20:32.628032 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:20:32.628005 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd7d751_21bf_4d99_bed4_a0e7a2d5e340.slice/crio-aa32065a982f9ea88c77759e4cd94f748a89dc7b83699aa7c06d3473504f1aa1 WatchSource:0}: Error finding container aa32065a982f9ea88c77759e4cd94f748a89dc7b83699aa7c06d3473504f1aa1: Status 404 returned error can't find the container with id aa32065a982f9ea88c77759e4cd94f748a89dc7b83699aa7c06d3473504f1aa1 Sep 11 14:20:32.630181 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.630167 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:20:32.803280 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:32.803233 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" event={"ID":"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340","Type":"ContainerStarted","Data":"aa32065a982f9ea88c77759e4cd94f748a89dc7b83699aa7c06d3473504f1aa1"} Sep 11 14:20:38.830379 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:38.830347 2578 generic.go:356] "Generic (PLEG): container finished" podID="1bd7d751-21bf-4d99-bed4-a0e7a2d5e340" containerID="3a08b94b5835ca258e6eb03133629e71696ebf1cbe01c5dab43195758d036483" exitCode=0 Sep 11 14:20:38.830761 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:38.830384 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" event={"ID":"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340","Type":"ContainerDied","Data":"3a08b94b5835ca258e6eb03133629e71696ebf1cbe01c5dab43195758d036483"} Sep 11 14:20:41.850811 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:41.850775 2578 generic.go:356] "Generic (PLEG): container finished" podID="1bd7d751-21bf-4d99-bed4-a0e7a2d5e340" containerID="9aacae930ea7508d88e557fb2f94b805f0339a61d6c187035471c91484a499de" exitCode=0 Sep 11 14:20:41.851262 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:41.850858 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" event={"ID":"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340","Type":"ContainerDied","Data":"9aacae930ea7508d88e557fb2f94b805f0339a61d6c187035471c91484a499de"} Sep 11 14:20:48.882343 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:48.882311 2578 generic.go:356] "Generic (PLEG): container finished" podID="1bd7d751-21bf-4d99-bed4-a0e7a2d5e340" containerID="53a9cf2246d8d8df2166cec8fda82dfa8e4cf8e9586a1df17ce8e8e4378f03a3" exitCode=0 Sep 11 14:20:48.882724 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:48.882398 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" event={"ID":"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340","Type":"ContainerDied","Data":"53a9cf2246d8d8df2166cec8fda82dfa8e4cf8e9586a1df17ce8e8e4378f03a3"} Sep 11 14:20:50.010154 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:50.010132 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" Sep 11 14:20:50.148994 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:50.148919 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-bundle\") pod \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\" (UID: \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\") " Sep 11 14:20:50.148994 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:50.148974 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-util\") pod \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\" (UID: \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\") " Sep 11 14:20:50.149201 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:50.149038 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdw7m\" (UniqueName: \"kubernetes.io/projected/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-kube-api-access-qdw7m\") pod \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\" (UID: \"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340\") " Sep 11 14:20:50.149578 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:50.149557 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-bundle" (OuterVolumeSpecName: "bundle") pod "1bd7d751-21bf-4d99-bed4-a0e7a2d5e340" (UID: "1bd7d751-21bf-4d99-bed4-a0e7a2d5e340"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:50.151174 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:50.151145 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-kube-api-access-qdw7m" (OuterVolumeSpecName: "kube-api-access-qdw7m") pod "1bd7d751-21bf-4d99-bed4-a0e7a2d5e340" (UID: "1bd7d751-21bf-4d99-bed4-a0e7a2d5e340"). InnerVolumeSpecName "kube-api-access-qdw7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:20:50.153279 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:50.153256 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-util" (OuterVolumeSpecName: "util") pod "1bd7d751-21bf-4d99-bed4-a0e7a2d5e340" (UID: "1bd7d751-21bf-4d99-bed4-a0e7a2d5e340"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:50.250163 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:50.250134 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:20:50.250163 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:50.250158 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdw7m\" (UniqueName: \"kubernetes.io/projected/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-kube-api-access-qdw7m\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:20:50.250304 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:50.250173 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1bd7d751-21bf-4d99-bed4-a0e7a2d5e340-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:20:50.891653 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:50.891631 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" Sep 11 14:20:50.891653 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:50.891640 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qkw4l" event={"ID":"1bd7d751-21bf-4d99-bed4-a0e7a2d5e340","Type":"ContainerDied","Data":"aa32065a982f9ea88c77759e4cd94f748a89dc7b83699aa7c06d3473504f1aa1"} Sep 11 14:20:50.891854 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:50.891678 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa32065a982f9ea88c77759e4cd94f748a89dc7b83699aa7c06d3473504f1aa1" Sep 11 14:20:59.713451 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.713415 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-74dzn"] Sep 11 14:20:59.713915 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.713790 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bd7d751-21bf-4d99-bed4-a0e7a2d5e340" containerName="util" Sep 11 14:20:59.713915 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.713803 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bd7d751-21bf-4d99-bed4-a0e7a2d5e340" containerName="util" Sep 11 14:20:59.713915 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.713813 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bd7d751-21bf-4d99-bed4-a0e7a2d5e340" containerName="pull" Sep 11 14:20:59.713915 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.713821 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bd7d751-21bf-4d99-bed4-a0e7a2d5e340" containerName="pull" Sep 11 14:20:59.713915 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.713835 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bd7d751-21bf-4d99-bed4-a0e7a2d5e340" containerName="extract" Sep 11 14:20:59.713915 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.713842 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bd7d751-21bf-4d99-bed4-a0e7a2d5e340" containerName="extract" Sep 11 14:20:59.713915 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.713905 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bd7d751-21bf-4d99-bed4-a0e7a2d5e340" containerName="extract" Sep 11 14:20:59.758735 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.758710 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-74dzn"] Sep 11 14:20:59.758891 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.758839 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-74dzn" Sep 11 14:20:59.761925 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.761902 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 14:20:59.763082 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.763062 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 14:20:59.763082 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.763076 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-mdnhw\"" Sep 11 14:20:59.921287 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.921250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsbjj\" (UniqueName: \"kubernetes.io/projected/d4d05f55-1e5c-4673-8308-46c2363f1a1e-kube-api-access-xsbjj\") pod \"cert-manager-webhook-6486ffd997-74dzn\" (UID: \"d4d05f55-1e5c-4673-8308-46c2363f1a1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-74dzn" Sep 11 14:20:59.921435 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:20:59.921339 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4d05f55-1e5c-4673-8308-46c2363f1a1e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-74dzn\" (UID: \"d4d05f55-1e5c-4673-8308-46c2363f1a1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-74dzn" Sep 11 14:21:00.022850 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.022763 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsbjj\" (UniqueName: \"kubernetes.io/projected/d4d05f55-1e5c-4673-8308-46c2363f1a1e-kube-api-access-xsbjj\") pod \"cert-manager-webhook-6486ffd997-74dzn\" (UID: \"d4d05f55-1e5c-4673-8308-46c2363f1a1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-74dzn" Sep 11 14:21:00.022850 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.022821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4d05f55-1e5c-4673-8308-46c2363f1a1e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-74dzn\" (UID: \"d4d05f55-1e5c-4673-8308-46c2363f1a1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-74dzn" Sep 11 14:21:00.036929 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.036904 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4d05f55-1e5c-4673-8308-46c2363f1a1e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-74dzn\" (UID: \"d4d05f55-1e5c-4673-8308-46c2363f1a1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-74dzn" Sep 11 14:21:00.037056 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.037036 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsbjj\" (UniqueName: \"kubernetes.io/projected/d4d05f55-1e5c-4673-8308-46c2363f1a1e-kube-api-access-xsbjj\") pod \"cert-manager-webhook-6486ffd997-74dzn\" (UID: \"d4d05f55-1e5c-4673-8308-46c2363f1a1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-74dzn" Sep 11 14:21:00.068005 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.067977 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-74dzn" Sep 11 14:21:00.400950 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.400926 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-74dzn"] Sep 11 14:21:00.403595 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:21:00.403569 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4d05f55_1e5c_4673_8308_46c2363f1a1e.slice/crio-a453a380a656eb73d5cd1c9148c0ede9e847432d80ddb2da879d19758bc474ba WatchSource:0}: Error finding container a453a380a656eb73d5cd1c9148c0ede9e847432d80ddb2da879d19758bc474ba: Status 404 returned error can't find the container with id a453a380a656eb73d5cd1c9148c0ede9e847432d80ddb2da879d19758bc474ba Sep 11 14:21:00.611663 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.611627 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8"] Sep 11 14:21:00.643764 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.643718 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8"] Sep 11 14:21:00.643925 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.643823 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" Sep 11 14:21:00.647923 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.647900 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8txz9\"" Sep 11 14:21:00.648058 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.647904 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:21:00.648058 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.647904 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:21:00.728582 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.728549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8\" (UID: \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" Sep 11 14:21:00.728947 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.728589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8\" (UID: \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" Sep 11 14:21:00.728947 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.728652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxplr\" (UniqueName: \"kubernetes.io/projected/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-kube-api-access-dxplr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8\" (UID: \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" Sep 11 14:21:00.829862 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.829830 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxplr\" (UniqueName: \"kubernetes.io/projected/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-kube-api-access-dxplr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8\" (UID: \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" Sep 11 14:21:00.830006 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.829896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8\" (UID: \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" Sep 11 14:21:00.830006 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.829917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8\" (UID: \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" Sep 11 14:21:00.830266 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.830243 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8\" (UID: \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" Sep 11 14:21:00.830334 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.830307 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8\" (UID: \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" Sep 11 14:21:00.841878 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.841854 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxplr\" (UniqueName: \"kubernetes.io/projected/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-kube-api-access-dxplr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8\" (UID: \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" Sep 11 14:21:00.929666 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.929598 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-74dzn" event={"ID":"d4d05f55-1e5c-4673-8308-46c2363f1a1e","Type":"ContainerStarted","Data":"a453a380a656eb73d5cd1c9148c0ede9e847432d80ddb2da879d19758bc474ba"} Sep 11 14:21:00.953956 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:00.953934 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" Sep 11 14:21:01.239331 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:01.239132 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8"] Sep 11 14:21:01.242307 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:21:01.242276 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c6fc71_19cc_4a39_b082_d4b6d43f3f6a.slice/crio-8f2f32b41206ce03ae9883cfeb2de6074bfe03882e63c20b08f842667313df3b WatchSource:0}: Error finding container 8f2f32b41206ce03ae9883cfeb2de6074bfe03882e63c20b08f842667313df3b: Status 404 returned error can't find the container with id 8f2f32b41206ce03ae9883cfeb2de6074bfe03882e63c20b08f842667313df3b Sep 11 14:21:01.937433 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:01.937398 2578 generic.go:356] "Generic (PLEG): container finished" podID="a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a" containerID="e01002dc5105fde36e2d587e859b6970e7224a5a8d288740be0f29002806c37a" exitCode=0 Sep 11 14:21:01.937925 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:01.937484 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" event={"ID":"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a","Type":"ContainerDied","Data":"e01002dc5105fde36e2d587e859b6970e7224a5a8d288740be0f29002806c37a"} Sep 11 14:21:01.937925 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:01.937552 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" event={"ID":"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a","Type":"ContainerStarted","Data":"8f2f32b41206ce03ae9883cfeb2de6074bfe03882e63c20b08f842667313df3b"} Sep 11 14:21:02.944105 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:02.944066 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-74dzn" event={"ID":"d4d05f55-1e5c-4673-8308-46c2363f1a1e","Type":"ContainerStarted","Data":"d00d4bb5f0a9dd4c536c42c623ec49e30fbcf26d90d034fd73426ddfdf5034c0"} Sep 11 14:21:02.944432 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:02.944323 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-74dzn" Sep 11 14:21:02.981113 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:02.981072 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-74dzn" podStartSLOduration=1.5146164 podStartE2EDuration="3.98105621s" totalImagesPullingTime="2.46643981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:21:02.978761957 +0000 UTC m=+351.064218314" watchObservedRunningTime="2026-09-11 14:21:02.98105621 +0000 UTC m=+351.066512571" Sep 11 14:21:03.177609 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:03.177530 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-v28qr"] Sep 11 14:21:03.180244 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:03.180215 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v28qr" Sep 11 14:21:03.183873 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:03.183828 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-888gl\"" Sep 11 14:21:03.204117 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:03.204092 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-v28qr"] Sep 11 14:21:03.352475 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:03.352437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gj7p\" (UniqueName: \"kubernetes.io/projected/1806261d-b85a-42a7-89bb-4761a153077b-kube-api-access-2gj7p\") pod \"cert-manager-cainjector-559cdd8c87-v28qr\" (UID: \"1806261d-b85a-42a7-89bb-4761a153077b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v28qr" Sep 11 14:21:03.352649 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:03.352560 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1806261d-b85a-42a7-89bb-4761a153077b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-v28qr\" (UID: \"1806261d-b85a-42a7-89bb-4761a153077b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v28qr" Sep 11 14:21:03.453620 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:03.453552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1806261d-b85a-42a7-89bb-4761a153077b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-v28qr\" (UID: \"1806261d-b85a-42a7-89bb-4761a153077b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v28qr" Sep 11 14:21:03.453782 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:03.453621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gj7p\" (UniqueName: \"kubernetes.io/projected/1806261d-b85a-42a7-89bb-4761a153077b-kube-api-access-2gj7p\") pod \"cert-manager-cainjector-559cdd8c87-v28qr\" (UID: \"1806261d-b85a-42a7-89bb-4761a153077b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v28qr" Sep 11 14:21:03.465088 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:03.465061 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1806261d-b85a-42a7-89bb-4761a153077b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-v28qr\" (UID: \"1806261d-b85a-42a7-89bb-4761a153077b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v28qr" Sep 11 14:21:03.465088 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:03.465082 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gj7p\" (UniqueName: \"kubernetes.io/projected/1806261d-b85a-42a7-89bb-4761a153077b-kube-api-access-2gj7p\") pod \"cert-manager-cainjector-559cdd8c87-v28qr\" (UID: \"1806261d-b85a-42a7-89bb-4761a153077b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v28qr" Sep 11 14:21:03.492028 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:03.492006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v28qr" Sep 11 14:21:04.450494 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:04.450472 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-v28qr"] Sep 11 14:21:04.452658 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:21:04.452622 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1806261d_b85a_42a7_89bb_4761a153077b.slice/crio-f520c127b7dd1335cdf5fcdee3c203ec26fc1790e337fda18804ac211dad0505 WatchSource:0}: Error finding container f520c127b7dd1335cdf5fcdee3c203ec26fc1790e337fda18804ac211dad0505: Status 404 returned error can't find the container with id f520c127b7dd1335cdf5fcdee3c203ec26fc1790e337fda18804ac211dad0505 Sep 11 14:21:04.955282 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:04.955206 2578 generic.go:356] "Generic (PLEG): container finished" podID="a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a" containerID="0f441337eb2e2f70d7b4245c6468aa743b5ac9d09346bf7f7f674231c269e7ee" exitCode=0 Sep 11 14:21:04.955452 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:04.955280 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" event={"ID":"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a","Type":"ContainerDied","Data":"0f441337eb2e2f70d7b4245c6468aa743b5ac9d09346bf7f7f674231c269e7ee"} Sep 11 14:21:04.957005 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:04.956980 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v28qr" event={"ID":"1806261d-b85a-42a7-89bb-4761a153077b","Type":"ContainerStarted","Data":"44b66c811a817dc9f4484bf05bd1c3093fb535679d3ebe7098a1ce01287463e8"} Sep 11 14:21:04.957112 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:04.957013 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v28qr" event={"ID":"1806261d-b85a-42a7-89bb-4761a153077b","Type":"ContainerStarted","Data":"f520c127b7dd1335cdf5fcdee3c203ec26fc1790e337fda18804ac211dad0505"} Sep 11 14:21:05.020517 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:05.020477 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v28qr" podStartSLOduration=2.020461469 podStartE2EDuration="2.020461469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:21:05.017980793 +0000 UTC m=+353.103437156" watchObservedRunningTime="2026-09-11 14:21:05.020461469 +0000 UTC m=+353.105917828" Sep 11 14:21:05.963596 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:05.963562 2578 generic.go:356] "Generic (PLEG): container finished" podID="a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a" containerID="9deb08fcaeede84d7ec4fba0fad02891e256090bd151fe37f72dfed0b0bed038" exitCode=0 Sep 11 14:21:05.964176 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:05.963653 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" event={"ID":"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a","Type":"ContainerDied","Data":"9deb08fcaeede84d7ec4fba0fad02891e256090bd151fe37f72dfed0b0bed038"} Sep 11 14:21:07.093486 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:07.093459 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" Sep 11 14:21:07.280713 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:07.280640 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxplr\" (UniqueName: \"kubernetes.io/projected/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-kube-api-access-dxplr\") pod \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\" (UID: \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\") " Sep 11 14:21:07.280713 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:07.280676 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-util\") pod \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\" (UID: \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\") " Sep 11 14:21:07.280897 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:07.280760 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-bundle\") pod \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\" (UID: \"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a\") " Sep 11 14:21:07.281169 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:07.281139 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-bundle" (OuterVolumeSpecName: "bundle") pod "a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a" (UID: "a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:07.282774 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:07.282751 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-kube-api-access-dxplr" (OuterVolumeSpecName: "kube-api-access-dxplr") pod "a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a" (UID: "a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a"). InnerVolumeSpecName "kube-api-access-dxplr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:21:07.284941 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:07.284909 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-util" (OuterVolumeSpecName: "util") pod "a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a" (UID: "a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:07.381765 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:07.381712 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:21:07.381907 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:07.381774 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxplr\" (UniqueName: \"kubernetes.io/projected/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-kube-api-access-dxplr\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:21:07.381907 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:07.381783 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:21:07.973050 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:07.973021 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" event={"ID":"a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a","Type":"ContainerDied","Data":"8f2f32b41206ce03ae9883cfeb2de6074bfe03882e63c20b08f842667313df3b"} Sep 11 14:21:07.973205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:07.973064 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f2f32b41206ce03ae9883cfeb2de6074bfe03882e63c20b08f842667313df3b" Sep 11 14:21:07.973205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:07.973067 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgndr8" Sep 11 14:21:08.951498 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:08.951466 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-74dzn" Sep 11 14:21:19.239060 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.239022 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l"] Sep 11 14:21:19.239510 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.239475 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a" containerName="pull" Sep 11 14:21:19.239510 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.239494 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a" containerName="pull" Sep 11 14:21:19.239621 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.239511 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a" containerName="util" Sep 11 14:21:19.239621 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.239520 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a" containerName="util" Sep 11 14:21:19.239621 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.239539 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a" containerName="extract" Sep 11 14:21:19.239621 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.239548 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a" containerName="extract" Sep 11 14:21:19.239870 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.239639 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0c6fc71-19cc-4a39-b082-d4b6d43f3f6a" containerName="extract" Sep 11 14:21:19.241908 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.241886 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" Sep 11 14:21:19.248532 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.248514 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:21:19.248603 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.248564 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:21:19.249898 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.249882 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8txz9\"" Sep 11 14:21:19.270831 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.270811 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/922b6ac3-8dc7-4769-8d27-614171e7949d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l\" (UID: \"922b6ac3-8dc7-4769-8d27-614171e7949d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" Sep 11 14:21:19.270926 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.270864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmgvg\" (UniqueName: \"kubernetes.io/projected/922b6ac3-8dc7-4769-8d27-614171e7949d-kube-api-access-wmgvg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l\" (UID: \"922b6ac3-8dc7-4769-8d27-614171e7949d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" Sep 11 14:21:19.270926 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.270910 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/922b6ac3-8dc7-4769-8d27-614171e7949d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l\" (UID: \"922b6ac3-8dc7-4769-8d27-614171e7949d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" Sep 11 14:21:19.272721 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.272701 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l"] Sep 11 14:21:19.371191 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.371165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmgvg\" (UniqueName: \"kubernetes.io/projected/922b6ac3-8dc7-4769-8d27-614171e7949d-kube-api-access-wmgvg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l\" (UID: \"922b6ac3-8dc7-4769-8d27-614171e7949d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" Sep 11 14:21:19.371299 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.371198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/922b6ac3-8dc7-4769-8d27-614171e7949d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l\" (UID: \"922b6ac3-8dc7-4769-8d27-614171e7949d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" Sep 11 14:21:19.371299 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.371232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/922b6ac3-8dc7-4769-8d27-614171e7949d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l\" (UID: \"922b6ac3-8dc7-4769-8d27-614171e7949d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" Sep 11 14:21:19.371537 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.371523 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/922b6ac3-8dc7-4769-8d27-614171e7949d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l\" (UID: \"922b6ac3-8dc7-4769-8d27-614171e7949d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" Sep 11 14:21:19.371587 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.371571 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/922b6ac3-8dc7-4769-8d27-614171e7949d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l\" (UID: \"922b6ac3-8dc7-4769-8d27-614171e7949d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" Sep 11 14:21:19.389332 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.389303 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmgvg\" (UniqueName: \"kubernetes.io/projected/922b6ac3-8dc7-4769-8d27-614171e7949d-kube-api-access-wmgvg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l\" (UID: \"922b6ac3-8dc7-4769-8d27-614171e7949d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" Sep 11 14:21:19.550094 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.550016 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" Sep 11 14:21:19.845396 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:19.845376 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l"] Sep 11 14:21:19.847503 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:21:19.847477 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922b6ac3_8dc7_4769_8d27_614171e7949d.slice/crio-2e5e539cfffdd6b5eaa3660287e03f7bc665907818a17d98db6b177dadb01d33 WatchSource:0}: Error finding container 2e5e539cfffdd6b5eaa3660287e03f7bc665907818a17d98db6b177dadb01d33: Status 404 returned error can't find the container with id 2e5e539cfffdd6b5eaa3660287e03f7bc665907818a17d98db6b177dadb01d33 Sep 11 14:21:20.025941 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:20.025906 2578 generic.go:356] "Generic (PLEG): container finished" podID="922b6ac3-8dc7-4769-8d27-614171e7949d" containerID="6170e73e81bdc4677c45199d75e259b7abbe5a8235b110e0dba136e9f88ddf33" exitCode=0 Sep 11 14:21:20.026063 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:20.025962 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" event={"ID":"922b6ac3-8dc7-4769-8d27-614171e7949d","Type":"ContainerDied","Data":"6170e73e81bdc4677c45199d75e259b7abbe5a8235b110e0dba136e9f88ddf33"} Sep 11 14:21:20.026063 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:20.026010 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" event={"ID":"922b6ac3-8dc7-4769-8d27-614171e7949d","Type":"ContainerStarted","Data":"2e5e539cfffdd6b5eaa3660287e03f7bc665907818a17d98db6b177dadb01d33"} Sep 11 14:21:21.032141 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:21.032106 2578 generic.go:356] "Generic (PLEG): container finished" podID="922b6ac3-8dc7-4769-8d27-614171e7949d" containerID="bf2ad1d1b9df94de781d64071bcf4bd54fe87043a78e651e9efeb8f005799079" exitCode=0 Sep 11 14:21:21.032554 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:21.032192 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" event={"ID":"922b6ac3-8dc7-4769-8d27-614171e7949d","Type":"ContainerDied","Data":"bf2ad1d1b9df94de781d64071bcf4bd54fe87043a78e651e9efeb8f005799079"} Sep 11 14:21:22.039217 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:22.039184 2578 generic.go:356] "Generic (PLEG): container finished" podID="922b6ac3-8dc7-4769-8d27-614171e7949d" containerID="69ce2316502c1232ab13e49ac18dae96ecb674848e5dcb172dc96aead92c6b19" exitCode=0 Sep 11 14:21:22.039656 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:22.039237 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" event={"ID":"922b6ac3-8dc7-4769-8d27-614171e7949d","Type":"ContainerDied","Data":"69ce2316502c1232ab13e49ac18dae96ecb674848e5dcb172dc96aead92c6b19"} Sep 11 14:21:23.161234 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:23.161210 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" Sep 11 14:21:23.199959 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:23.199934 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/922b6ac3-8dc7-4769-8d27-614171e7949d-bundle\") pod \"922b6ac3-8dc7-4769-8d27-614171e7949d\" (UID: \"922b6ac3-8dc7-4769-8d27-614171e7949d\") " Sep 11 14:21:23.200075 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:23.199968 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/922b6ac3-8dc7-4769-8d27-614171e7949d-util\") pod \"922b6ac3-8dc7-4769-8d27-614171e7949d\" (UID: \"922b6ac3-8dc7-4769-8d27-614171e7949d\") " Sep 11 14:21:23.200075 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:23.200011 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmgvg\" (UniqueName: \"kubernetes.io/projected/922b6ac3-8dc7-4769-8d27-614171e7949d-kube-api-access-wmgvg\") pod \"922b6ac3-8dc7-4769-8d27-614171e7949d\" (UID: \"922b6ac3-8dc7-4769-8d27-614171e7949d\") " Sep 11 14:21:23.200692 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:23.200667 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922b6ac3-8dc7-4769-8d27-614171e7949d-bundle" (OuterVolumeSpecName: "bundle") pod "922b6ac3-8dc7-4769-8d27-614171e7949d" (UID: "922b6ac3-8dc7-4769-8d27-614171e7949d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:23.202045 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:23.202014 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922b6ac3-8dc7-4769-8d27-614171e7949d-kube-api-access-wmgvg" (OuterVolumeSpecName: "kube-api-access-wmgvg") pod "922b6ac3-8dc7-4769-8d27-614171e7949d" (UID: "922b6ac3-8dc7-4769-8d27-614171e7949d"). InnerVolumeSpecName "kube-api-access-wmgvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:21:23.208829 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:23.208809 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922b6ac3-8dc7-4769-8d27-614171e7949d-util" (OuterVolumeSpecName: "util") pod "922b6ac3-8dc7-4769-8d27-614171e7949d" (UID: "922b6ac3-8dc7-4769-8d27-614171e7949d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:23.300846 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:23.300782 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/922b6ac3-8dc7-4769-8d27-614171e7949d-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:21:23.300846 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:23.300805 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/922b6ac3-8dc7-4769-8d27-614171e7949d-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:21:23.300846 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:23.300815 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmgvg\" (UniqueName: \"kubernetes.io/projected/922b6ac3-8dc7-4769-8d27-614171e7949d-kube-api-access-wmgvg\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:21:24.049884 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:24.049853 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" event={"ID":"922b6ac3-8dc7-4769-8d27-614171e7949d","Type":"ContainerDied","Data":"2e5e539cfffdd6b5eaa3660287e03f7bc665907818a17d98db6b177dadb01d33"} Sep 11 14:21:24.049884 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:24.049881 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqcv4l" Sep 11 14:21:24.050090 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:24.049901 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e5e539cfffdd6b5eaa3660287e03f7bc665907818a17d98db6b177dadb01d33" Sep 11 14:21:32.908753 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.908700 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n"] Sep 11 14:21:32.909229 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.909163 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922b6ac3-8dc7-4769-8d27-614171e7949d" containerName="pull" Sep 11 14:21:32.909229 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.909180 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="922b6ac3-8dc7-4769-8d27-614171e7949d" containerName="pull" Sep 11 14:21:32.909229 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.909192 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922b6ac3-8dc7-4769-8d27-614171e7949d" containerName="util" Sep 11 14:21:32.909229 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.909201 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="922b6ac3-8dc7-4769-8d27-614171e7949d" containerName="util" Sep 11 14:21:32.909436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.909236 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922b6ac3-8dc7-4769-8d27-614171e7949d" containerName="extract" Sep 11 14:21:32.909436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.909244 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="922b6ac3-8dc7-4769-8d27-614171e7949d" containerName="extract" Sep 11 14:21:32.909436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.909350 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="922b6ac3-8dc7-4769-8d27-614171e7949d" containerName="extract" Sep 11 14:21:32.913543 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.913524 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" Sep 11 14:21:32.916749 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.916708 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:21:32.916832 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.916786 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8txz9\"" Sep 11 14:21:32.918029 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.918009 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:21:32.929166 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.929136 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n"] Sep 11 14:21:32.977555 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.977532 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4t22\" (UniqueName: \"kubernetes.io/projected/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-kube-api-access-c4t22\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n\" (UID: \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" Sep 11 14:21:32.977643 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.977574 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n\" (UID: \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" Sep 11 14:21:32.977703 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:32.977682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n\" (UID: \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" Sep 11 14:21:33.078197 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:33.078159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n\" (UID: \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" Sep 11 14:21:33.078347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:33.078203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4t22\" (UniqueName: \"kubernetes.io/projected/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-kube-api-access-c4t22\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n\" (UID: \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" Sep 11 14:21:33.078424 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:33.078404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n\" (UID: \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" Sep 11 14:21:33.078559 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:33.078537 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n\" (UID: \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" Sep 11 14:21:33.078677 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:33.078661 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n\" (UID: \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" Sep 11 14:21:33.094215 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:33.094185 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4t22\" (UniqueName: \"kubernetes.io/projected/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-kube-api-access-c4t22\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n\" (UID: \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" Sep 11 14:21:33.222442 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:33.222353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" Sep 11 14:21:33.508350 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:33.508323 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n"] Sep 11 14:21:33.509642 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:21:33.509616 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9a1fb4_ce49_4c7c_bb2a_4b1691310a7f.slice/crio-d81df759aece37ac7834c6fa5c2cda9601b1b352052df2f848f32d6edaf56b1e WatchSource:0}: Error finding container d81df759aece37ac7834c6fa5c2cda9601b1b352052df2f848f32d6edaf56b1e: Status 404 returned error can't find the container with id d81df759aece37ac7834c6fa5c2cda9601b1b352052df2f848f32d6edaf56b1e Sep 11 14:21:34.093482 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.093448 2578 generic.go:356] "Generic (PLEG): container finished" podID="2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f" containerID="a05cd016ccbc829e3a61c9cf4442f0476143b3866b51bffd646135fd0d067302" exitCode=0 Sep 11 14:21:34.093949 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.093537 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" event={"ID":"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f","Type":"ContainerDied","Data":"a05cd016ccbc829e3a61c9cf4442f0476143b3866b51bffd646135fd0d067302"} Sep 11 14:21:34.093949 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.093583 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" event={"ID":"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f","Type":"ContainerStarted","Data":"d81df759aece37ac7834c6fa5c2cda9601b1b352052df2f848f32d6edaf56b1e"} Sep 11 14:21:34.170805 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.170771 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49"] Sep 11 14:21:34.174197 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.174182 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.179763 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.179709 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 14:21:34.179763 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.179713 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 14:21:34.179967 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.179840 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-srflq\"" Sep 11 14:21:34.184165 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.184148 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 14:21:34.184352 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.184339 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 14:21:34.208372 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.208347 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49"] Sep 11 14:21:34.287512 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.287481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfjv9\" (UniqueName: \"kubernetes.io/projected/911aa1fb-0413-429e-8a25-dfe6615e528c-kube-api-access-mfjv9\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.287669 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.287563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911aa1fb-0413-429e-8a25-dfe6615e528c-tmp\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.287669 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.287607 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/911aa1fb-0413-429e-8a25-dfe6615e528c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.287669 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.287638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/911aa1fb-0413-429e-8a25-dfe6615e528c-manifests\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.287669 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.287652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/911aa1fb-0413-429e-8a25-dfe6615e528c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.388928 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.388899 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911aa1fb-0413-429e-8a25-dfe6615e528c-tmp\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.389107 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.388935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/911aa1fb-0413-429e-8a25-dfe6615e528c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.389107 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.388958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/911aa1fb-0413-429e-8a25-dfe6615e528c-manifests\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.389107 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.388974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/911aa1fb-0413-429e-8a25-dfe6615e528c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.389107 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.389007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfjv9\" (UniqueName: \"kubernetes.io/projected/911aa1fb-0413-429e-8a25-dfe6615e528c-kube-api-access-mfjv9\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.389383 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.389363 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/911aa1fb-0413-429e-8a25-dfe6615e528c-manifests\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.391359 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.391336 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911aa1fb-0413-429e-8a25-dfe6615e528c-tmp\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.391559 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.391540 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/911aa1fb-0413-429e-8a25-dfe6615e528c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.391634 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.391540 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/911aa1fb-0413-429e-8a25-dfe6615e528c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.407685 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.407661 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfjv9\" (UniqueName: \"kubernetes.io/projected/911aa1fb-0413-429e-8a25-dfe6615e528c-kube-api-access-mfjv9\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-j8f49\" (UID: \"911aa1fb-0413-429e-8a25-dfe6615e528c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.484989 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.484956 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:34.810090 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:34.810059 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49"] Sep 11 14:21:34.813031 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:21:34.812999 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911aa1fb_0413_429e_8a25_dfe6615e528c.slice/crio-b66bee0988e12a4eabe2465493b9782990cdaa6490985f3323061184c2f16dff WatchSource:0}: Error finding container b66bee0988e12a4eabe2465493b9782990cdaa6490985f3323061184c2f16dff: Status 404 returned error can't find the container with id b66bee0988e12a4eabe2465493b9782990cdaa6490985f3323061184c2f16dff Sep 11 14:21:35.100572 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:35.100535 2578 generic.go:356] "Generic (PLEG): container finished" podID="2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f" containerID="b0b25a8b96f11b6b19e69ae58158136bc2483468082b7e4d2b0bfd3b8c5052a2" exitCode=0 Sep 11 14:21:35.101047 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:35.100614 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" event={"ID":"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f","Type":"ContainerDied","Data":"b0b25a8b96f11b6b19e69ae58158136bc2483468082b7e4d2b0bfd3b8c5052a2"} Sep 11 14:21:35.102338 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:35.102318 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" event={"ID":"911aa1fb-0413-429e-8a25-dfe6615e528c","Type":"ContainerStarted","Data":"b66bee0988e12a4eabe2465493b9782990cdaa6490985f3323061184c2f16dff"} Sep 11 14:21:36.110896 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:36.110856 2578 generic.go:356] "Generic (PLEG): container finished" podID="2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f" containerID="19c2dfed73620a49a1ece156647f69ac6eddb8dd727884d4c7c1b57a1bc046b5" exitCode=0 Sep 11 14:21:36.111352 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:36.110939 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" event={"ID":"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f","Type":"ContainerDied","Data":"19c2dfed73620a49a1ece156647f69ac6eddb8dd727884d4c7c1b57a1bc046b5"} Sep 11 14:21:37.843441 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:37.843412 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" Sep 11 14:21:37.919880 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:37.919852 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-util\") pod \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\" (UID: \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\") " Sep 11 14:21:37.920044 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:37.919920 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-bundle\") pod \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\" (UID: \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\") " Sep 11 14:21:37.920044 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:37.919947 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4t22\" (UniqueName: \"kubernetes.io/projected/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-kube-api-access-c4t22\") pod \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\" (UID: \"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f\") " Sep 11 14:21:37.920633 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:37.920607 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-bundle" (OuterVolumeSpecName: "bundle") pod "2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f" (UID: "2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:37.922046 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:37.922015 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-kube-api-access-c4t22" (OuterVolumeSpecName: "kube-api-access-c4t22") pod "2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f" (UID: "2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f"). InnerVolumeSpecName "kube-api-access-c4t22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:21:37.925306 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:37.925271 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-util" (OuterVolumeSpecName: "util") pod "2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f" (UID: "2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:21:38.021236 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:38.021199 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:21:38.021236 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:38.021229 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:21:38.021236 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:38.021241 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4t22\" (UniqueName: \"kubernetes.io/projected/2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f-kube-api-access-c4t22\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:21:38.124507 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:38.124480 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" event={"ID":"2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f","Type":"ContainerDied","Data":"d81df759aece37ac7834c6fa5c2cda9601b1b352052df2f848f32d6edaf56b1e"} Sep 11 14:21:38.124627 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:38.124514 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pr88n" Sep 11 14:21:38.124627 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:38.124529 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d81df759aece37ac7834c6fa5c2cda9601b1b352052df2f848f32d6edaf56b1e" Sep 11 14:21:39.131162 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:39.131129 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" event={"ID":"911aa1fb-0413-429e-8a25-dfe6615e528c","Type":"ContainerStarted","Data":"e6d341d5d08fcf80cc8c97389f08b30f7960e3f042e28b68a49e0a24fa958882"} Sep 11 14:21:40.137081 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:40.137048 2578 generic.go:356] "Generic (PLEG): container finished" podID="911aa1fb-0413-429e-8a25-dfe6615e528c" containerID="e6d341d5d08fcf80cc8c97389f08b30f7960e3f042e28b68a49e0a24fa958882" exitCode=0 Sep 11 14:21:40.137457 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:40.137123 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" event={"ID":"911aa1fb-0413-429e-8a25-dfe6615e528c","Type":"ContainerDied","Data":"e6d341d5d08fcf80cc8c97389f08b30f7960e3f042e28b68a49e0a24fa958882"} Sep 11 14:21:41.143367 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:41.143331 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" event={"ID":"911aa1fb-0413-429e-8a25-dfe6615e528c","Type":"ContainerStarted","Data":"a07546a9697689bbd5732c6531b2c7a2b8d380f596ca78c5f8fc99b00752489f"} Sep 11 14:21:41.143759 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:41.143467 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:52.150210 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:52.150179 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" Sep 11 14:21:52.187327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:52.187280 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-j8f49" podStartSLOduration=13.477210481 podStartE2EDuration="18.187262868s" totalImagesPullingTime="3.647491077s" totalInitContainerRuntime="1.06256131s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:21:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:21:41.211027911 +0000 UTC m=+389.296484270" watchObservedRunningTime="2026-09-11 14:21:52.187262868 +0000 UTC m=+400.272719228" Sep 11 14:21:54.484454 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.484418 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l"] Sep 11 14:21:54.484903 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.484791 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f" containerName="pull" Sep 11 14:21:54.484903 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.484806 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f" containerName="pull" Sep 11 14:21:54.484903 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.484830 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f" containerName="extract" Sep 11 14:21:54.484903 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.484853 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f" containerName="extract" Sep 11 14:21:54.484903 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.484860 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f" containerName="util" Sep 11 14:21:54.484903 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.484865 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f" containerName="util" Sep 11 14:21:54.485109 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.484930 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a9a1fb4-ce49-4c7c-bb2a-4b1691310a7f" containerName="extract" Sep 11 14:21:54.492418 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.492400 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" Sep 11 14:21:54.502628 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.502609 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-crxlp\"" Sep 11 14:21:54.505460 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.505442 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 14:21:54.505633 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.505621 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 14:21:54.510240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.510220 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l"] Sep 11 14:21:54.551168 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.551141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knc5k\" (UniqueName: \"kubernetes.io/projected/c1db64bb-87ce-4241-909f-5965e92f7a25-kube-api-access-knc5k\") pod \"kube-auth-proxy-57f955fb8f-kkv8l\" (UID: \"c1db64bb-87ce-4241-909f-5965e92f7a25\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" Sep 11 14:21:54.551311 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.551186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1db64bb-87ce-4241-909f-5965e92f7a25-tmp\") pod \"kube-auth-proxy-57f955fb8f-kkv8l\" (UID: \"c1db64bb-87ce-4241-909f-5965e92f7a25\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" Sep 11 14:21:54.551311 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.551290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1db64bb-87ce-4241-909f-5965e92f7a25-tls-certs\") pod \"kube-auth-proxy-57f955fb8f-kkv8l\" (UID: \"c1db64bb-87ce-4241-909f-5965e92f7a25\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" Sep 11 14:21:54.652447 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.652420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knc5k\" (UniqueName: \"kubernetes.io/projected/c1db64bb-87ce-4241-909f-5965e92f7a25-kube-api-access-knc5k\") pod \"kube-auth-proxy-57f955fb8f-kkv8l\" (UID: \"c1db64bb-87ce-4241-909f-5965e92f7a25\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" Sep 11 14:21:54.652562 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.652455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1db64bb-87ce-4241-909f-5965e92f7a25-tmp\") pod \"kube-auth-proxy-57f955fb8f-kkv8l\" (UID: \"c1db64bb-87ce-4241-909f-5965e92f7a25\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" Sep 11 14:21:54.652664 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.652639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1db64bb-87ce-4241-909f-5965e92f7a25-tls-certs\") pod \"kube-auth-proxy-57f955fb8f-kkv8l\" (UID: \"c1db64bb-87ce-4241-909f-5965e92f7a25\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" Sep 11 14:21:54.654657 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.654628 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1db64bb-87ce-4241-909f-5965e92f7a25-tmp\") pod \"kube-auth-proxy-57f955fb8f-kkv8l\" (UID: \"c1db64bb-87ce-4241-909f-5965e92f7a25\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" Sep 11 14:21:54.654927 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.654909 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1db64bb-87ce-4241-909f-5965e92f7a25-tls-certs\") pod \"kube-auth-proxy-57f955fb8f-kkv8l\" (UID: \"c1db64bb-87ce-4241-909f-5965e92f7a25\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" Sep 11 14:21:54.680017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.679999 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knc5k\" (UniqueName: \"kubernetes.io/projected/c1db64bb-87ce-4241-909f-5965e92f7a25-kube-api-access-knc5k\") pod \"kube-auth-proxy-57f955fb8f-kkv8l\" (UID: \"c1db64bb-87ce-4241-909f-5965e92f7a25\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" Sep 11 14:21:54.800801 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:54.800707 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" Sep 11 14:21:55.094471 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:55.094446 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l"] Sep 11 14:21:55.096394 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:21:55.096366 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1db64bb_87ce_4241_909f_5965e92f7a25.slice/crio-39a55604621973f2863bf5f7b1586f1c31853964923845b80e373092367382da WatchSource:0}: Error finding container 39a55604621973f2863bf5f7b1586f1c31853964923845b80e373092367382da: Status 404 returned error can't find the container with id 39a55604621973f2863bf5f7b1586f1c31853964923845b80e373092367382da Sep 11 14:21:55.207973 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:55.207935 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" event={"ID":"c1db64bb-87ce-4241-909f-5965e92f7a25","Type":"ContainerStarted","Data":"39a55604621973f2863bf5f7b1586f1c31853964923845b80e373092367382da"} Sep 11 14:21:57.440946 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.440911 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8"] Sep 11 14:21:57.444894 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.444867 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.450709 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.450681 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 14:21:57.450709 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.450684 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 14:21:57.451174 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.451158 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 14:21:57.451773 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.451754 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 14:21:57.451886 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.451829 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lf2rd\"" Sep 11 14:21:57.460119 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.460096 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8"] Sep 11 14:21:57.587151 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.587108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.587317 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.587166 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.587317 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.587224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4p8m\" (UniqueName: \"kubernetes.io/projected/b56b1f64-ba0a-46d6-8726-adf34d3856e2-kube-api-access-j4p8m\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.587317 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.587294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.587489 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.587338 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56b1f64-ba0a-46d6-8726-adf34d3856e2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.587489 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.587393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.587489 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.587425 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.688692 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.688656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.688874 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.688704 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.688874 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.688770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.688874 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.688796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.689019 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.688997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4p8m\" (UniqueName: \"kubernetes.io/projected/b56b1f64-ba0a-46d6-8726-adf34d3856e2-kube-api-access-j4p8m\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.689077 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.689056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.689133 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.689115 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56b1f64-ba0a-46d6-8726-adf34d3856e2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.689766 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.689720 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.691680 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.691623 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.691680 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.691664 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56b1f64-ba0a-46d6-8726-adf34d3856e2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.691867 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.691666 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.691986 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.691965 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.710346 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.710188 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4p8m\" (UniqueName: \"kubernetes.io/projected/b56b1f64-ba0a-46d6-8726-adf34d3856e2-kube-api-access-j4p8m\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.711111 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.711050 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-nvhs8\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:57.756084 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:57.756054 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:21:58.120123 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:58.120096 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8"] Sep 11 14:21:58.121539 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:21:58.121513 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56b1f64_ba0a_46d6_8726_adf34d3856e2.slice/crio-44bca7d62bb970c11ae48ebd041540a4ae1ba903f242bbf791f21580089ea814 WatchSource:0}: Error finding container 44bca7d62bb970c11ae48ebd041540a4ae1ba903f242bbf791f21580089ea814: Status 404 returned error can't find the container with id 44bca7d62bb970c11ae48ebd041540a4ae1ba903f242bbf791f21580089ea814 Sep 11 14:21:58.224970 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:58.224889 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" event={"ID":"b56b1f64-ba0a-46d6-8726-adf34d3856e2","Type":"ContainerStarted","Data":"44bca7d62bb970c11ae48ebd041540a4ae1ba903f242bbf791f21580089ea814"} Sep 11 14:21:58.226301 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:58.226275 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" event={"ID":"c1db64bb-87ce-4241-909f-5965e92f7a25","Type":"ContainerStarted","Data":"72d3270894b291caba1231fdd7dbbf9ccad160044b5096b7641e3ebc9d5a6c2b"} Sep 11 14:21:58.295809 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:21:58.295752 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57f955fb8f-kkv8l" podStartSLOduration=1.492765885 podStartE2EDuration="4.295711s" totalImagesPullingTime="2.802945115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:21:58.295588669 +0000 UTC m=+406.381045043" watchObservedRunningTime="2026-09-11 14:21:58.295711 +0000 UTC m=+406.381167360" Sep 11 14:22:00.632664 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:00.632626 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:22:00.632958 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:00.632693 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:22:01.244029 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:01.243987 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" event={"ID":"b56b1f64-ba0a-46d6-8726-adf34d3856e2","Type":"ContainerStarted","Data":"12ceeff7f556403f95fe3f7837cca8d070f6c8da826ca46f95f3528b737fc835"} Sep 11 14:22:01.244217 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:01.244171 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:22:01.246108 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:01.246083 2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-nvhs8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:22:01.246218 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:01.246137 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" podUID="b56b1f64-ba0a-46d6-8726-adf34d3856e2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:22:02.248832 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:02.248798 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:22:02.283953 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:02.283904 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" podStartSLOduration=2.775018708 podStartE2EDuration="5.283887324s" totalImagesPullingTime="2.508868616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:21:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:01.27548102 +0000 UTC m=+409.360937374" watchObservedRunningTime="2026-09-11 14:22:02.283887324 +0000 UTC m=+410.369343684" Sep 11 14:22:18.118219 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.118179 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz"] Sep 11 14:22:18.127996 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.127969 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" Sep 11 14:22:18.129906 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.129881 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz"] Sep 11 14:22:18.133160 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.133136 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" Sep 11 14:22:18.136439 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.135807 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 11 14:22:18.136439 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.135982 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz"] Sep 11 14:22:18.137484 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.137461 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 11 14:22:18.137587 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.137493 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-zrb8c\"" Sep 11 14:22:18.139098 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.139077 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-78hbf\"" Sep 11 14:22:18.156607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.156578 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz"] Sep 11 14:22:18.160162 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.160133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5nw9\" (UniqueName: \"kubernetes.io/projected/c245cb01-c3b2-4e88-9958-8987f62d91a8-kube-api-access-d5nw9\") pod \"kserve-controller-manager-d6f4b65b8-pb8jz\" (UID: \"c245cb01-c3b2-4e88-9958-8987f62d91a8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" Sep 11 14:22:18.160295 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.160174 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebc9d097-340d-4a73-ba10-1b716ff4b7c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dk4tz\" (UID: \"ebc9d097-340d-4a73-ba10-1b716ff4b7c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" Sep 11 14:22:18.160295 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.160244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c245cb01-c3b2-4e88-9958-8987f62d91a8-cert\") pod \"kserve-controller-manager-d6f4b65b8-pb8jz\" (UID: \"c245cb01-c3b2-4e88-9958-8987f62d91a8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" Sep 11 14:22:18.160371 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.160322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62dt6\" (UniqueName: \"kubernetes.io/projected/ebc9d097-340d-4a73-ba10-1b716ff4b7c0-kube-api-access-62dt6\") pod \"llmisvc-controller-manager-777c6bd7d5-dk4tz\" (UID: \"ebc9d097-340d-4a73-ba10-1b716ff4b7c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" Sep 11 14:22:18.261434 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.261397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62dt6\" (UniqueName: \"kubernetes.io/projected/ebc9d097-340d-4a73-ba10-1b716ff4b7c0-kube-api-access-62dt6\") pod \"llmisvc-controller-manager-777c6bd7d5-dk4tz\" (UID: \"ebc9d097-340d-4a73-ba10-1b716ff4b7c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" Sep 11 14:22:18.261603 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.261451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5nw9\" (UniqueName: \"kubernetes.io/projected/c245cb01-c3b2-4e88-9958-8987f62d91a8-kube-api-access-d5nw9\") pod \"kserve-controller-manager-d6f4b65b8-pb8jz\" (UID: \"c245cb01-c3b2-4e88-9958-8987f62d91a8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" Sep 11 14:22:18.261603 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.261468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebc9d097-340d-4a73-ba10-1b716ff4b7c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dk4tz\" (UID: \"ebc9d097-340d-4a73-ba10-1b716ff4b7c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" Sep 11 14:22:18.261603 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.261505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c245cb01-c3b2-4e88-9958-8987f62d91a8-cert\") pod \"kserve-controller-manager-d6f4b65b8-pb8jz\" (UID: \"c245cb01-c3b2-4e88-9958-8987f62d91a8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" Sep 11 14:22:18.261803 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:22:18.261635 2578 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 11 14:22:18.261803 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:22:18.261715 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc9d097-340d-4a73-ba10-1b716ff4b7c0-cert podName:ebc9d097-340d-4a73-ba10-1b716ff4b7c0 nodeName:}" failed. No retries permitted until 2026-09-11 14:22:18.761693877 +0000 UTC m=+426.847150233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ebc9d097-340d-4a73-ba10-1b716ff4b7c0-cert") pod "llmisvc-controller-manager-777c6bd7d5-dk4tz" (UID: "ebc9d097-340d-4a73-ba10-1b716ff4b7c0") : secret "llmisvc-webhook-server-cert" not found Sep 11 14:22:18.263956 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.263934 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c245cb01-c3b2-4e88-9958-8987f62d91a8-cert\") pod \"kserve-controller-manager-d6f4b65b8-pb8jz\" (UID: \"c245cb01-c3b2-4e88-9958-8987f62d91a8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" Sep 11 14:22:18.294849 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.294821 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5nw9\" (UniqueName: \"kubernetes.io/projected/c245cb01-c3b2-4e88-9958-8987f62d91a8-kube-api-access-d5nw9\") pod \"kserve-controller-manager-d6f4b65b8-pb8jz\" (UID: \"c245cb01-c3b2-4e88-9958-8987f62d91a8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" Sep 11 14:22:18.295120 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.295098 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62dt6\" (UniqueName: \"kubernetes.io/projected/ebc9d097-340d-4a73-ba10-1b716ff4b7c0-kube-api-access-62dt6\") pod \"llmisvc-controller-manager-777c6bd7d5-dk4tz\" (UID: \"ebc9d097-340d-4a73-ba10-1b716ff4b7c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" Sep 11 14:22:18.441154 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.441078 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" Sep 11 14:22:18.749030 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:22:18.748995 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc245cb01_c3b2_4e88_9958_8987f62d91a8.slice/crio-693486e5984779e4ea6dc96e5acf407a34a214133134772d92071e77192c8cc0 WatchSource:0}: Error finding container 693486e5984779e4ea6dc96e5acf407a34a214133134772d92071e77192c8cc0: Status 404 returned error can't find the container with id 693486e5984779e4ea6dc96e5acf407a34a214133134772d92071e77192c8cc0 Sep 11 14:22:18.753980 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.753957 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz"] Sep 11 14:22:18.765087 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.765063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebc9d097-340d-4a73-ba10-1b716ff4b7c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dk4tz\" (UID: \"ebc9d097-340d-4a73-ba10-1b716ff4b7c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" Sep 11 14:22:18.767680 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:18.767650 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ebc9d097-340d-4a73-ba10-1b716ff4b7c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dk4tz\" (UID: \"ebc9d097-340d-4a73-ba10-1b716ff4b7c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" Sep 11 14:22:19.047178 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:19.047089 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" Sep 11 14:22:19.331718 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:19.331633 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" event={"ID":"c245cb01-c3b2-4e88-9958-8987f62d91a8","Type":"ContainerStarted","Data":"693486e5984779e4ea6dc96e5acf407a34a214133134772d92071e77192c8cc0"} Sep 11 14:22:19.340972 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:19.340948 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz"] Sep 11 14:22:19.342860 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:22:19.342835 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podebc9d097_340d_4a73_ba10_1b716ff4b7c0.slice/crio-bc77e6f03f76091a7e2deae7d1f5992b40b7b42b95194aafa2fc3406bf6d847d WatchSource:0}: Error finding container bc77e6f03f76091a7e2deae7d1f5992b40b7b42b95194aafa2fc3406bf6d847d: Status 404 returned error can't find the container with id bc77e6f03f76091a7e2deae7d1f5992b40b7b42b95194aafa2fc3406bf6d847d Sep 11 14:22:19.875084 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:19.875048 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8"] Sep 11 14:22:19.875312 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:19.875284 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" podUID="b56b1f64-ba0a-46d6-8726-adf34d3856e2" containerName="discovery" containerID="cri-o://12ceeff7f556403f95fe3f7837cca8d070f6c8da826ca46f95f3528b737fc835" gracePeriod=30 Sep 11 14:22:20.338808 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.338679 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" event={"ID":"ebc9d097-340d-4a73-ba10-1b716ff4b7c0","Type":"ContainerStarted","Data":"bc77e6f03f76091a7e2deae7d1f5992b40b7b42b95194aafa2fc3406bf6d847d"} Sep 11 14:22:20.340709 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.340675 2578 generic.go:356] "Generic (PLEG): container finished" podID="b56b1f64-ba0a-46d6-8726-adf34d3856e2" containerID="12ceeff7f556403f95fe3f7837cca8d070f6c8da826ca46f95f3528b737fc835" exitCode=0 Sep 11 14:22:20.340860 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.340742 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" event={"ID":"b56b1f64-ba0a-46d6-8726-adf34d3856e2","Type":"ContainerDied","Data":"12ceeff7f556403f95fe3f7837cca8d070f6c8da826ca46f95f3528b737fc835"} Sep 11 14:22:20.647536 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.647510 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:22:20.680813 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.680790 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-token\") pod \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " Sep 11 14:22:20.680915 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.680821 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56b1f64-ba0a-46d6-8726-adf34d3856e2-local-certs\") pod \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " Sep 11 14:22:20.680915 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.680851 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4p8m\" (UniqueName: \"kubernetes.io/projected/b56b1f64-ba0a-46d6-8726-adf34d3856e2-kube-api-access-j4p8m\") pod \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " Sep 11 14:22:20.680915 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.680872 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-csr-ca-configmap\") pod \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " Sep 11 14:22:20.680915 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.680900 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-cacerts\") pod \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " Sep 11 14:22:20.681132 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.680923 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-kubeconfig\") pod \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " Sep 11 14:22:20.681132 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.680940 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-csr-dns-cert\") pod \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\" (UID: \"b56b1f64-ba0a-46d6-8726-adf34d3856e2\") " Sep 11 14:22:20.681697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.681645 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b56b1f64-ba0a-46d6-8726-adf34d3856e2" (UID: "b56b1f64-ba0a-46d6-8726-adf34d3856e2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:22:20.683334 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.683294 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b56b1f64-ba0a-46d6-8726-adf34d3856e2" (UID: "b56b1f64-ba0a-46d6-8726-adf34d3856e2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:22:20.683444 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.683330 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56b1f64-ba0a-46d6-8726-adf34d3856e2-kube-api-access-j4p8m" (OuterVolumeSpecName: "kube-api-access-j4p8m") pod "b56b1f64-ba0a-46d6-8726-adf34d3856e2" (UID: "b56b1f64-ba0a-46d6-8726-adf34d3856e2"). InnerVolumeSpecName "kube-api-access-j4p8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:22:20.683444 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.683387 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56b1f64-ba0a-46d6-8726-adf34d3856e2-local-certs" (OuterVolumeSpecName: "local-certs") pod "b56b1f64-ba0a-46d6-8726-adf34d3856e2" (UID: "b56b1f64-ba0a-46d6-8726-adf34d3856e2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:22:20.683444 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.683434 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-cacerts" (OuterVolumeSpecName: "cacerts") pod "b56b1f64-ba0a-46d6-8726-adf34d3856e2" (UID: "b56b1f64-ba0a-46d6-8726-adf34d3856e2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:22:20.683600 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.683448 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-token" (OuterVolumeSpecName: "istio-token") pod "b56b1f64-ba0a-46d6-8726-adf34d3856e2" (UID: "b56b1f64-ba0a-46d6-8726-adf34d3856e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:22:20.683600 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.683520 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b56b1f64-ba0a-46d6-8726-adf34d3856e2" (UID: "b56b1f64-ba0a-46d6-8726-adf34d3856e2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:22:20.781552 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.781519 2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-kubeconfig\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:22:20.781552 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.781549 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-csr-dns-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:22:20.781552 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.781560 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-token\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:22:20.781790 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.781568 2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56b1f64-ba0a-46d6-8726-adf34d3856e2-local-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:22:20.781790 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.781581 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4p8m\" (UniqueName: \"kubernetes.io/projected/b56b1f64-ba0a-46d6-8726-adf34d3856e2-kube-api-access-j4p8m\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:22:20.781790 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.781594 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b56b1f64-ba0a-46d6-8726-adf34d3856e2-istio-csr-ca-configmap\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:22:20.781790 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:20.781604 2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b56b1f64-ba0a-46d6-8726-adf34d3856e2-cacerts\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:22:21.348981 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:21.348941 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" event={"ID":"c245cb01-c3b2-4e88-9958-8987f62d91a8","Type":"ContainerStarted","Data":"05bec0f39043c1ce58c62082f46ed1f579f4864a69ef0e4daf86fe61f7c3a991"} Sep 11 14:22:21.349442 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:21.349046 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" Sep 11 14:22:21.350565 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:21.350524 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" event={"ID":"b56b1f64-ba0a-46d6-8726-adf34d3856e2","Type":"ContainerDied","Data":"44bca7d62bb970c11ae48ebd041540a4ae1ba903f242bbf791f21580089ea814"} Sep 11 14:22:21.350565 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:21.350556 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" Sep 11 14:22:21.350781 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:21.350583 2578 scope.go:117] "RemoveContainer" containerID="12ceeff7f556403f95fe3f7837cca8d070f6c8da826ca46f95f3528b737fc835" Sep 11 14:22:21.381470 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:21.381417 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" podStartSLOduration=1.452885717 podStartE2EDuration="3.38139645s" totalImagesPullingTime="1.928510733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:21.379704605 +0000 UTC m=+429.465160967" watchObservedRunningTime="2026-09-11 14:22:21.38139645 +0000 UTC m=+429.466852845" Sep 11 14:22:21.413572 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:21.413515 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8"] Sep 11 14:22:21.416560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:21.416526 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8"] Sep 11 14:22:22.464581 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:22.464549 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b56b1f64-ba0a-46d6-8726-adf34d3856e2" path="/var/lib/kubelet/pods/b56b1f64-ba0a-46d6-8726-adf34d3856e2/volumes" Sep 11 14:22:23.364798 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:23.364767 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" event={"ID":"ebc9d097-340d-4a73-ba10-1b716ff4b7c0","Type":"ContainerStarted","Data":"4aefc9485bab8c5128caf1ed8343b05726949cf609bddc910fc9bc47915ba9c4"} Sep 11 14:22:23.364977 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:23.364881 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" Sep 11 14:22:23.398333 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:23.398286 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" podStartSLOduration=1.864772432 podStartE2EDuration="5.39827219s" totalImagesPullingTime="3.533499758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:23.394180583 +0000 UTC m=+431.479636943" watchObservedRunningTime="2026-09-11 14:22:23.39827219 +0000 UTC m=+431.483728580" Sep 11 14:22:25.249351 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:25.249311 2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-nvhs8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:22:25.249818 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:25.249366 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-nvhs8" podUID="b56b1f64-ba0a-46d6-8726-adf34d3856e2" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.31:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:22:52.364407 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:52.364381 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" Sep 11 14:22:54.371836 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:22:54.371799 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" Sep 11 14:23:10.993123 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:10.993090 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z"] Sep 11 14:23:10.993488 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:10.993436 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56b1f64-ba0a-46d6-8726-adf34d3856e2" containerName="discovery" Sep 11 14:23:10.993488 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:10.993448 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56b1f64-ba0a-46d6-8726-adf34d3856e2" containerName="discovery" Sep 11 14:23:10.993612 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:10.993505 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56b1f64-ba0a-46d6-8726-adf34d3856e2" containerName="discovery" Sep 11 14:23:10.996090 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:10.996074 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" Sep 11 14:23:11.000114 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.000097 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:23:11.000619 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.000601 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:23:11.001659 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.001642 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8txz9\"" Sep 11 14:23:11.013663 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.013640 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z"] Sep 11 14:23:11.104420 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.104389 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn"] Sep 11 14:23:11.107507 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.107490 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" Sep 11 14:23:11.122686 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.122665 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn"] Sep 11 14:23:11.177093 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.177066 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7mzk\" (UniqueName: \"kubernetes.io/projected/e47a8fc0-1900-42ba-bed8-7c2a6765db97-kube-api-access-c7mzk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z\" (UID: \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" Sep 11 14:23:11.177233 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.177105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e47a8fc0-1900-42ba-bed8-7c2a6765db97-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z\" (UID: \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" Sep 11 14:23:11.177307 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.177251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e47a8fc0-1900-42ba-bed8-7c2a6765db97-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z\" (UID: \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" Sep 11 14:23:11.187281 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.187255 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8"] Sep 11 14:23:11.190317 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.190301 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" Sep 11 14:23:11.203682 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.203664 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8"] Sep 11 14:23:11.277973 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.277900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1df1f955-be84-4b07-a9c4-475ef12a2d4f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn\" (UID: \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" Sep 11 14:23:11.277973 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.277943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e47a8fc0-1900-42ba-bed8-7c2a6765db97-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z\" (UID: \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" Sep 11 14:23:11.277973 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.277966 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8\" (UID: \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" Sep 11 14:23:11.278196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.278028 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v5sj\" (UniqueName: \"kubernetes.io/projected/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-kube-api-access-4v5sj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8\" (UID: \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" Sep 11 14:23:11.278196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.278060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1df1f955-be84-4b07-a9c4-475ef12a2d4f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn\" (UID: \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" Sep 11 14:23:11.278196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.278082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzsm5\" (UniqueName: \"kubernetes.io/projected/1df1f955-be84-4b07-a9c4-475ef12a2d4f-kube-api-access-hzsm5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn\" (UID: \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" Sep 11 14:23:11.278343 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.278198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mzk\" (UniqueName: \"kubernetes.io/projected/e47a8fc0-1900-42ba-bed8-7c2a6765db97-kube-api-access-c7mzk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z\" (UID: \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" Sep 11 14:23:11.278343 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.278249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e47a8fc0-1900-42ba-bed8-7c2a6765db97-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z\" (UID: \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" Sep 11 14:23:11.278343 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.278284 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e47a8fc0-1900-42ba-bed8-7c2a6765db97-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z\" (UID: \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" Sep 11 14:23:11.278343 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.278283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8\" (UID: \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" Sep 11 14:23:11.278593 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.278574 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e47a8fc0-1900-42ba-bed8-7c2a6765db97-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z\" (UID: \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" Sep 11 14:23:11.294054 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.294002 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7"] Sep 11 14:23:11.295955 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.295933 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mzk\" (UniqueName: \"kubernetes.io/projected/e47a8fc0-1900-42ba-bed8-7c2a6765db97-kube-api-access-c7mzk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z\" (UID: \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" Sep 11 14:23:11.303314 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.303298 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" Sep 11 14:23:11.305300 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.305286 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" Sep 11 14:23:11.310291 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.310265 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7"] Sep 11 14:23:11.379709 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.379676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8\" (UID: \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" Sep 11 14:23:11.379855 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.379798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v5sj\" (UniqueName: \"kubernetes.io/projected/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-kube-api-access-4v5sj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8\" (UID: \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" Sep 11 14:23:11.379855 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.379840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1df1f955-be84-4b07-a9c4-475ef12a2d4f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn\" (UID: \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" Sep 11 14:23:11.379992 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.379870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsm5\" (UniqueName: \"kubernetes.io/projected/1df1f955-be84-4b07-a9c4-475ef12a2d4f-kube-api-access-hzsm5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn\" (UID: \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" Sep 11 14:23:11.379992 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.379919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8\" (UID: \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" Sep 11 14:23:11.379992 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.379984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1df1f955-be84-4b07-a9c4-475ef12a2d4f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn\" (UID: \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" Sep 11 14:23:11.380511 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.380336 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1df1f955-be84-4b07-a9c4-475ef12a2d4f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn\" (UID: \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" Sep 11 14:23:11.380511 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.380409 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1df1f955-be84-4b07-a9c4-475ef12a2d4f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn\" (UID: \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" Sep 11 14:23:11.380511 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.380478 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8\" (UID: \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" Sep 11 14:23:11.380511 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.380504 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8\" (UID: \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" Sep 11 14:23:11.395166 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.395149 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v5sj\" (UniqueName: \"kubernetes.io/projected/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-kube-api-access-4v5sj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8\" (UID: \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" Sep 11 14:23:11.395885 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.395871 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzsm5\" (UniqueName: \"kubernetes.io/projected/1df1f955-be84-4b07-a9c4-475ef12a2d4f-kube-api-access-hzsm5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn\" (UID: \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" Sep 11 14:23:11.416443 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.416424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" Sep 11 14:23:11.481221 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.481194 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7\" (UID: \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" Sep 11 14:23:11.481339 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.481239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7\" (UID: \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" Sep 11 14:23:11.481339 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.481262 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbmq6\" (UniqueName: \"kubernetes.io/projected/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-kube-api-access-xbmq6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7\" (UID: \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" Sep 11 14:23:11.498207 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.498183 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" Sep 11 14:23:11.582502 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.582479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7\" (UID: \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" Sep 11 14:23:11.582628 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.582515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7\" (UID: \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" Sep 11 14:23:11.582628 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.582548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbmq6\" (UniqueName: \"kubernetes.io/projected/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-kube-api-access-xbmq6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7\" (UID: \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" Sep 11 14:23:11.582938 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.582911 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7\" (UID: \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" Sep 11 14:23:11.583040 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.582925 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7\" (UID: \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" Sep 11 14:23:11.588912 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.588892 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z"] Sep 11 14:23:11.590149 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:23:11.590124 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47a8fc0_1900_42ba_bed8_7c2a6765db97.slice/crio-2516b99799a4ab062530afac958d8de2ccab05be331044645e49a575fde334b4 WatchSource:0}: Error finding container 2516b99799a4ab062530afac958d8de2ccab05be331044645e49a575fde334b4: Status 404 returned error can't find the container with id 2516b99799a4ab062530afac958d8de2ccab05be331044645e49a575fde334b4 Sep 11 14:23:11.594431 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.594410 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbmq6\" (UniqueName: \"kubernetes.io/projected/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-kube-api-access-xbmq6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7\" (UID: \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" Sep 11 14:23:11.617398 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.617376 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" Sep 11 14:23:11.696296 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.696269 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn"] Sep 11 14:23:11.739477 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:23:11.739455 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df1f955_be84_4b07_a9c4_475ef12a2d4f.slice/crio-e3de8810e43c798bce43c9f3822f81572be7f4d329a9601892a10d126d9a2e9c WatchSource:0}: Error finding container e3de8810e43c798bce43c9f3822f81572be7f4d329a9601892a10d126d9a2e9c: Status 404 returned error can't find the container with id e3de8810e43c798bce43c9f3822f81572be7f4d329a9601892a10d126d9a2e9c Sep 11 14:23:11.778380 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.778354 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8"] Sep 11 14:23:11.786776 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:23:11.786701 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd1d62d_e0e2_4e15_ab0c_ab0eb6bb96ff.slice/crio-60fb906e0c086a20916ccf6e7345b20b290e522ee04f959c3356eb547326d625 WatchSource:0}: Error finding container 60fb906e0c086a20916ccf6e7345b20b290e522ee04f959c3356eb547326d625: Status 404 returned error can't find the container with id 60fb906e0c086a20916ccf6e7345b20b290e522ee04f959c3356eb547326d625 Sep 11 14:23:11.897042 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:11.897020 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7"] Sep 11 14:23:11.933462 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:23:11.933429 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc2b4e4_0e3b_4b71_b43d_b664d71de9f3.slice/crio-2f578f7634413c462258304f6a98291535f79cde5e7b1214b77dcdb8897730f6 WatchSource:0}: Error finding container 2f578f7634413c462258304f6a98291535f79cde5e7b1214b77dcdb8897730f6: Status 404 returned error can't find the container with id 2f578f7634413c462258304f6a98291535f79cde5e7b1214b77dcdb8897730f6 Sep 11 14:23:12.591704 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:12.591619 2578 generic.go:356] "Generic (PLEG): container finished" podID="1df1f955-be84-4b07-a9c4-475ef12a2d4f" containerID="32774f0ea474e12ef27650da9e75c729ca856d2a9f11620ec7bbe27664f39b2c" exitCode=0 Sep 11 14:23:12.592146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:12.591701 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" event={"ID":"1df1f955-be84-4b07-a9c4-475ef12a2d4f","Type":"ContainerDied","Data":"32774f0ea474e12ef27650da9e75c729ca856d2a9f11620ec7bbe27664f39b2c"} Sep 11 14:23:12.592146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:12.591777 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" event={"ID":"1df1f955-be84-4b07-a9c4-475ef12a2d4f","Type":"ContainerStarted","Data":"e3de8810e43c798bce43c9f3822f81572be7f4d329a9601892a10d126d9a2e9c"} Sep 11 14:23:12.593375 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:12.593352 2578 generic.go:356] "Generic (PLEG): container finished" podID="1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff" containerID="00f416735ca3154efb211bfe22d851970ac0c187abd12691a080c4287422562f" exitCode=0 Sep 11 14:23:12.593488 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:12.593380 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" event={"ID":"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff","Type":"ContainerDied","Data":"00f416735ca3154efb211bfe22d851970ac0c187abd12691a080c4287422562f"} Sep 11 14:23:12.593488 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:12.593421 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" event={"ID":"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff","Type":"ContainerStarted","Data":"60fb906e0c086a20916ccf6e7345b20b290e522ee04f959c3356eb547326d625"} Sep 11 14:23:12.595028 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:12.595009 2578 generic.go:356] "Generic (PLEG): container finished" podID="e47a8fc0-1900-42ba-bed8-7c2a6765db97" containerID="dc7f4b5cbe28254004267bf30797c3a3019fc5591d36cbf7eb7d506a5c2236bf" exitCode=0 Sep 11 14:23:12.595110 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:12.595092 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" event={"ID":"e47a8fc0-1900-42ba-bed8-7c2a6765db97","Type":"ContainerDied","Data":"dc7f4b5cbe28254004267bf30797c3a3019fc5591d36cbf7eb7d506a5c2236bf"} Sep 11 14:23:12.595163 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:12.595139 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" event={"ID":"e47a8fc0-1900-42ba-bed8-7c2a6765db97","Type":"ContainerStarted","Data":"2516b99799a4ab062530afac958d8de2ccab05be331044645e49a575fde334b4"} Sep 11 14:23:12.596562 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:12.596539 2578 generic.go:356] "Generic (PLEG): container finished" podID="8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3" containerID="fd26185809dd8d0b7a0ca37d0b5373b5096f285cb65c3093864e27152370b5e4" exitCode=0 Sep 11 14:23:12.596648 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:12.596584 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" event={"ID":"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3","Type":"ContainerDied","Data":"fd26185809dd8d0b7a0ca37d0b5373b5096f285cb65c3093864e27152370b5e4"} Sep 11 14:23:12.596648 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:12.596633 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" event={"ID":"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3","Type":"ContainerStarted","Data":"2f578f7634413c462258304f6a98291535f79cde5e7b1214b77dcdb8897730f6"} Sep 11 14:23:13.603003 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:13.602969 2578 generic.go:356] "Generic (PLEG): container finished" podID="1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff" containerID="a1b5056a07193548133f7d3dfcb7e38e7a6db5b609a9e4d35ac1325cd2b00a85" exitCode=0 Sep 11 14:23:13.603347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:13.603049 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" event={"ID":"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff","Type":"ContainerDied","Data":"a1b5056a07193548133f7d3dfcb7e38e7a6db5b609a9e4d35ac1325cd2b00a85"} Sep 11 14:23:13.605134 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:13.605106 2578 generic.go:356] "Generic (PLEG): container finished" podID="e47a8fc0-1900-42ba-bed8-7c2a6765db97" containerID="7238f3142d7ea079ff71b29b0b291323f6ccab5dc25a07d5482ebb9886014660" exitCode=0 Sep 11 14:23:13.605242 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:13.605202 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" event={"ID":"e47a8fc0-1900-42ba-bed8-7c2a6765db97","Type":"ContainerDied","Data":"7238f3142d7ea079ff71b29b0b291323f6ccab5dc25a07d5482ebb9886014660"} Sep 11 14:23:13.607298 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:13.607278 2578 generic.go:356] "Generic (PLEG): container finished" podID="1df1f955-be84-4b07-a9c4-475ef12a2d4f" containerID="d7b7399faf44c1f34e590af6e1b519e0bc1b8ab570df3c79ebb0989023c67c46" exitCode=0 Sep 11 14:23:13.607373 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:13.607308 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" event={"ID":"1df1f955-be84-4b07-a9c4-475ef12a2d4f","Type":"ContainerDied","Data":"d7b7399faf44c1f34e590af6e1b519e0bc1b8ab570df3c79ebb0989023c67c46"} Sep 11 14:23:14.615356 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:14.615320 2578 generic.go:356] "Generic (PLEG): container finished" podID="1df1f955-be84-4b07-a9c4-475ef12a2d4f" containerID="914e0578583ed6e2b5f247f96f147168ae3d70835ae611d79c854353b1ee517d" exitCode=0 Sep 11 14:23:14.615751 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:14.615399 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" event={"ID":"1df1f955-be84-4b07-a9c4-475ef12a2d4f","Type":"ContainerDied","Data":"914e0578583ed6e2b5f247f96f147168ae3d70835ae611d79c854353b1ee517d"} Sep 11 14:23:14.617397 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:14.617372 2578 generic.go:356] "Generic (PLEG): container finished" podID="1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff" containerID="8a6a1c7ec106c06fb7c9367fee2b53658432b5a18e82fa1d1c41eb4cdf3ae64b" exitCode=0 Sep 11 14:23:14.617506 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:14.617439 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" event={"ID":"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff","Type":"ContainerDied","Data":"8a6a1c7ec106c06fb7c9367fee2b53658432b5a18e82fa1d1c41eb4cdf3ae64b"} Sep 11 14:23:14.619271 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:14.619252 2578 generic.go:356] "Generic (PLEG): container finished" podID="e47a8fc0-1900-42ba-bed8-7c2a6765db97" containerID="dc975f371d0f9743d4660f92540ecf3eadaf8ae48083aa3b1db9dca237b8c311" exitCode=0 Sep 11 14:23:14.619352 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:14.619333 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" event={"ID":"e47a8fc0-1900-42ba-bed8-7c2a6765db97","Type":"ContainerDied","Data":"dc975f371d0f9743d4660f92540ecf3eadaf8ae48083aa3b1db9dca237b8c311"} Sep 11 14:23:14.621022 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:14.621002 2578 generic.go:356] "Generic (PLEG): container finished" podID="8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3" containerID="889347a57526a6dd2ddeace3291183ec1dc850d58ab188a08b637e52828a36c3" exitCode=0 Sep 11 14:23:14.621096 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:14.621062 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" event={"ID":"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3","Type":"ContainerDied","Data":"889347a57526a6dd2ddeace3291183ec1dc850d58ab188a08b637e52828a36c3"} Sep 11 14:23:15.628222 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.628190 2578 generic.go:356] "Generic (PLEG): container finished" podID="8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3" containerID="adc31c7602016f5b8ed6e50fdbf755481cbeeea344dd715e65855c6b80d9711c" exitCode=0 Sep 11 14:23:15.628594 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.628279 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" event={"ID":"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3","Type":"ContainerDied","Data":"adc31c7602016f5b8ed6e50fdbf755481cbeeea344dd715e65855c6b80d9711c"} Sep 11 14:23:15.805564 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.805540 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" Sep 11 14:23:15.808742 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.808710 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" Sep 11 14:23:15.812001 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.811984 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" Sep 11 14:23:15.914187 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.914111 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1df1f955-be84-4b07-a9c4-475ef12a2d4f-bundle\") pod \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\" (UID: \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\") " Sep 11 14:23:15.914187 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.914139 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-util\") pod \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\" (UID: \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\") " Sep 11 14:23:15.914187 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.914159 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v5sj\" (UniqueName: \"kubernetes.io/projected/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-kube-api-access-4v5sj\") pod \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\" (UID: \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\") " Sep 11 14:23:15.914445 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.914192 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7mzk\" (UniqueName: \"kubernetes.io/projected/e47a8fc0-1900-42ba-bed8-7c2a6765db97-kube-api-access-c7mzk\") pod \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\" (UID: \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\") " Sep 11 14:23:15.914445 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.914212 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-bundle\") pod \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\" (UID: \"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff\") " Sep 11 14:23:15.914445 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.914226 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1df1f955-be84-4b07-a9c4-475ef12a2d4f-util\") pod \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\" (UID: \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\") " Sep 11 14:23:15.914445 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.914278 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e47a8fc0-1900-42ba-bed8-7c2a6765db97-util\") pod \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\" (UID: \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\") " Sep 11 14:23:15.914445 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.914327 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzsm5\" (UniqueName: \"kubernetes.io/projected/1df1f955-be84-4b07-a9c4-475ef12a2d4f-kube-api-access-hzsm5\") pod \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\" (UID: \"1df1f955-be84-4b07-a9c4-475ef12a2d4f\") " Sep 11 14:23:15.914445 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.914368 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e47a8fc0-1900-42ba-bed8-7c2a6765db97-bundle\") pod \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\" (UID: \"e47a8fc0-1900-42ba-bed8-7c2a6765db97\") " Sep 11 14:23:15.915142 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.914905 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df1f955-be84-4b07-a9c4-475ef12a2d4f-bundle" (OuterVolumeSpecName: "bundle") pod "1df1f955-be84-4b07-a9c4-475ef12a2d4f" (UID: "1df1f955-be84-4b07-a9c4-475ef12a2d4f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:15.915453 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.915428 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47a8fc0-1900-42ba-bed8-7c2a6765db97-bundle" (OuterVolumeSpecName: "bundle") pod "e47a8fc0-1900-42ba-bed8-7c2a6765db97" (UID: "e47a8fc0-1900-42ba-bed8-7c2a6765db97"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:15.915546 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.915513 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-bundle" (OuterVolumeSpecName: "bundle") pod "1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff" (UID: "1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:15.916883 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.916854 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47a8fc0-1900-42ba-bed8-7c2a6765db97-kube-api-access-c7mzk" (OuterVolumeSpecName: "kube-api-access-c7mzk") pod "e47a8fc0-1900-42ba-bed8-7c2a6765db97" (UID: "e47a8fc0-1900-42ba-bed8-7c2a6765db97"). InnerVolumeSpecName "kube-api-access-c7mzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:15.917004 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.916894 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-kube-api-access-4v5sj" (OuterVolumeSpecName: "kube-api-access-4v5sj") pod "1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff" (UID: "1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff"). InnerVolumeSpecName "kube-api-access-4v5sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:15.917519 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.917488 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df1f955-be84-4b07-a9c4-475ef12a2d4f-kube-api-access-hzsm5" (OuterVolumeSpecName: "kube-api-access-hzsm5") pod "1df1f955-be84-4b07-a9c4-475ef12a2d4f" (UID: "1df1f955-be84-4b07-a9c4-475ef12a2d4f"). InnerVolumeSpecName "kube-api-access-hzsm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:15.921301 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.921278 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-util" (OuterVolumeSpecName: "util") pod "1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff" (UID: "1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:15.921885 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.921864 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df1f955-be84-4b07-a9c4-475ef12a2d4f-util" (OuterVolumeSpecName: "util") pod "1df1f955-be84-4b07-a9c4-475ef12a2d4f" (UID: "1df1f955-be84-4b07-a9c4-475ef12a2d4f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:15.923949 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:15.923920 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47a8fc0-1900-42ba-bed8-7c2a6765db97-util" (OuterVolumeSpecName: "util") pod "e47a8fc0-1900-42ba-bed8-7c2a6765db97" (UID: "e47a8fc0-1900-42ba-bed8-7c2a6765db97"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:16.015056 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.015032 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e47a8fc0-1900-42ba-bed8-7c2a6765db97-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:16.015056 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.015055 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzsm5\" (UniqueName: \"kubernetes.io/projected/1df1f955-be84-4b07-a9c4-475ef12a2d4f-kube-api-access-hzsm5\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:16.015201 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.015065 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e47a8fc0-1900-42ba-bed8-7c2a6765db97-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:16.015201 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.015074 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1df1f955-be84-4b07-a9c4-475ef12a2d4f-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:16.015201 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.015081 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:16.015201 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.015089 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v5sj\" (UniqueName: \"kubernetes.io/projected/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-kube-api-access-4v5sj\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:16.015201 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.015097 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7mzk\" (UniqueName: \"kubernetes.io/projected/e47a8fc0-1900-42ba-bed8-7c2a6765db97-kube-api-access-c7mzk\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:16.015201 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.015106 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:16.015201 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.015113 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1df1f955-be84-4b07-a9c4-475ef12a2d4f-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:16.635388 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.635363 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" Sep 11 14:23:16.635712 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.635364 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7zzsn" event={"ID":"1df1f955-be84-4b07-a9c4-475ef12a2d4f","Type":"ContainerDied","Data":"e3de8810e43c798bce43c9f3822f81572be7f4d329a9601892a10d126d9a2e9c"} Sep 11 14:23:16.635712 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.635482 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3de8810e43c798bce43c9f3822f81572be7f4d329a9601892a10d126d9a2e9c" Sep 11 14:23:16.637491 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.637471 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" event={"ID":"1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff","Type":"ContainerDied","Data":"60fb906e0c086a20916ccf6e7345b20b290e522ee04f959c3356eb547326d625"} Sep 11 14:23:16.637491 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.637486 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jgdg8" Sep 11 14:23:16.637635 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.637503 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60fb906e0c086a20916ccf6e7345b20b290e522ee04f959c3356eb547326d625" Sep 11 14:23:16.639625 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.639608 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" Sep 11 14:23:16.639625 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.639621 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahqb6z" event={"ID":"e47a8fc0-1900-42ba-bed8-7c2a6765db97","Type":"ContainerDied","Data":"2516b99799a4ab062530afac958d8de2ccab05be331044645e49a575fde334b4"} Sep 11 14:23:16.639757 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.639660 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2516b99799a4ab062530afac958d8de2ccab05be331044645e49a575fde334b4" Sep 11 14:23:16.754431 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.754404 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" Sep 11 14:23:16.924537 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.924509 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbmq6\" (UniqueName: \"kubernetes.io/projected/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-kube-api-access-xbmq6\") pod \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\" (UID: \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\") " Sep 11 14:23:16.924670 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.924571 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-bundle\") pod \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\" (UID: \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\") " Sep 11 14:23:16.924670 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.924601 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-util\") pod \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\" (UID: \"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3\") " Sep 11 14:23:16.925240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.925202 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-bundle" (OuterVolumeSpecName: "bundle") pod "8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3" (UID: "8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:16.926593 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.926569 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-kube-api-access-xbmq6" (OuterVolumeSpecName: "kube-api-access-xbmq6") pod "8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3" (UID: "8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3"). InnerVolumeSpecName "kube-api-access-xbmq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:16.929881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:16.929844 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-util" (OuterVolumeSpecName: "util") pod "8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3" (UID: "8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:17.025845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:17.025781 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-bundle\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:17.025845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:17.025805 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-util\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:17.025845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:17.025815 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbmq6\" (UniqueName: \"kubernetes.io/projected/8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3-kube-api-access-xbmq6\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:17.646807 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:17.646780 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" Sep 11 14:23:17.647164 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:17.646779 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ntft7" event={"ID":"8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3","Type":"ContainerDied","Data":"2f578f7634413c462258304f6a98291535f79cde5e7b1214b77dcdb8897730f6"} Sep 11 14:23:17.647164 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:17.646895 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f578f7634413c462258304f6a98291535f79cde5e7b1214b77dcdb8897730f6" Sep 11 14:23:23.900682 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.900641 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-9h7sk"] Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901148 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e47a8fc0-1900-42ba-bed8-7c2a6765db97" containerName="util" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901169 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e47a8fc0-1900-42ba-bed8-7c2a6765db97" containerName="util" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901182 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3" containerName="pull" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901191 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3" containerName="pull" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901210 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3" containerName="util" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901221 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3" containerName="util" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901233 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1df1f955-be84-4b07-a9c4-475ef12a2d4f" containerName="util" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901241 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1df1f955-be84-4b07-a9c4-475ef12a2d4f" containerName="util" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901253 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff" containerName="pull" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901261 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff" containerName="pull" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901270 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1df1f955-be84-4b07-a9c4-475ef12a2d4f" containerName="pull" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901278 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1df1f955-be84-4b07-a9c4-475ef12a2d4f" containerName="pull" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901286 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e47a8fc0-1900-42ba-bed8-7c2a6765db97" containerName="extract" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901295 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e47a8fc0-1900-42ba-bed8-7c2a6765db97" containerName="extract" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901308 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff" containerName="util" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901316 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff" containerName="util" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901325 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff" containerName="extract" Sep 11 14:23:23.901327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901333 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff" containerName="extract" Sep 11 14:23:23.902223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901341 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1df1f955-be84-4b07-a9c4-475ef12a2d4f" containerName="extract" Sep 11 14:23:23.902223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901349 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1df1f955-be84-4b07-a9c4-475ef12a2d4f" containerName="extract" Sep 11 14:23:23.902223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901357 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3" containerName="extract" Sep 11 14:23:23.902223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901363 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3" containerName="extract" Sep 11 14:23:23.902223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901381 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e47a8fc0-1900-42ba-bed8-7c2a6765db97" containerName="pull" Sep 11 14:23:23.902223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901389 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e47a8fc0-1900-42ba-bed8-7c2a6765db97" containerName="pull" Sep 11 14:23:23.902223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901473 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc2b4e4-0e3b-4b71-b43d-b664d71de9f3" containerName="extract" Sep 11 14:23:23.902223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901487 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fd1d62d-e0e2-4e15-ab0c-ab0eb6bb96ff" containerName="extract" Sep 11 14:23:23.902223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901498 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1df1f955-be84-4b07-a9c4-475ef12a2d4f" containerName="extract" Sep 11 14:23:23.902223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.901510 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e47a8fc0-1900-42ba-bed8-7c2a6765db97" containerName="extract" Sep 11 14:23:23.906136 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.906117 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" Sep 11 14:23:23.912675 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.912659 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-69dll\"" Sep 11 14:23:23.926704 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.926680 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 14:23:23.926837 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.926741 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 14:23:23.952349 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.952317 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-9h7sk"] Sep 11 14:23:23.984971 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:23.984939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5ff\" (UniqueName: \"kubernetes.io/projected/c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb-kube-api-access-fd5ff\") pod \"authorino-operator-76b478bd78-9h7sk\" (UID: \"c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb\") " pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" Sep 11 14:23:24.085361 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:24.085330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5ff\" (UniqueName: \"kubernetes.io/projected/c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb-kube-api-access-fd5ff\") pod \"authorino-operator-76b478bd78-9h7sk\" (UID: \"c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb\") " pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" Sep 11 14:23:24.099537 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:24.099512 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5ff\" (UniqueName: \"kubernetes.io/projected/c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb-kube-api-access-fd5ff\") pod \"authorino-operator-76b478bd78-9h7sk\" (UID: \"c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb\") " pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" Sep 11 14:23:24.216211 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:24.216134 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" Sep 11 14:23:24.516133 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:24.516065 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-9h7sk"] Sep 11 14:23:24.519401 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:23:24.519376 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d8e2c2_9fe3_4728_82a0_fc4ea5faaacb.slice/crio-85d46cdb45b93a3b81bf7e7e6bd89e3802457bebda7ac411b7b0f86f370887bd WatchSource:0}: Error finding container 85d46cdb45b93a3b81bf7e7e6bd89e3802457bebda7ac411b7b0f86f370887bd: Status 404 returned error can't find the container with id 85d46cdb45b93a3b81bf7e7e6bd89e3802457bebda7ac411b7b0f86f370887bd Sep 11 14:23:24.686078 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:24.686048 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" event={"ID":"c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb","Type":"ContainerStarted","Data":"85d46cdb45b93a3b81bf7e7e6bd89e3802457bebda7ac411b7b0f86f370887bd"} Sep 11 14:23:26.039797 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:26.039759 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt"] Sep 11 14:23:26.048946 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:26.048917 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" Sep 11 14:23:26.052135 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:26.052105 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 11 14:23:26.052551 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:26.052480 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-w9479\"" Sep 11 14:23:26.076313 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:26.076247 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt"] Sep 11 14:23:26.101876 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:26.101844 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgbqj\" (UniqueName: \"kubernetes.io/projected/23e76c7f-aeae-4756-bb38-eba9de65e417-kube-api-access-dgbqj\") pod \"dns-operator-controller-manager-7b84cd7c9c-7m8nt\" (UID: \"23e76c7f-aeae-4756-bb38-eba9de65e417\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" Sep 11 14:23:26.203763 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:26.203348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgbqj\" (UniqueName: \"kubernetes.io/projected/23e76c7f-aeae-4756-bb38-eba9de65e417-kube-api-access-dgbqj\") pod \"dns-operator-controller-manager-7b84cd7c9c-7m8nt\" (UID: \"23e76c7f-aeae-4756-bb38-eba9de65e417\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" Sep 11 14:23:26.237548 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:26.237485 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgbqj\" (UniqueName: \"kubernetes.io/projected/23e76c7f-aeae-4756-bb38-eba9de65e417-kube-api-access-dgbqj\") pod \"dns-operator-controller-manager-7b84cd7c9c-7m8nt\" (UID: \"23e76c7f-aeae-4756-bb38-eba9de65e417\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" Sep 11 14:23:26.364460 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:26.364415 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" Sep 11 14:23:26.668165 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:26.668138 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt"] Sep 11 14:23:27.466777 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:23:27.466723 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e76c7f_aeae_4756_bb38_eba9de65e417.slice/crio-1d525c93dfb3143058cbc8ce6a3c8916bd8d50e2e291203b3dd6a0047cf758a5 WatchSource:0}: Error finding container 1d525c93dfb3143058cbc8ce6a3c8916bd8d50e2e291203b3dd6a0047cf758a5: Status 404 returned error can't find the container with id 1d525c93dfb3143058cbc8ce6a3c8916bd8d50e2e291203b3dd6a0047cf758a5 Sep 11 14:23:27.702305 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:27.702273 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" event={"ID":"23e76c7f-aeae-4756-bb38-eba9de65e417","Type":"ContainerStarted","Data":"1d525c93dfb3143058cbc8ce6a3c8916bd8d50e2e291203b3dd6a0047cf758a5"} Sep 11 14:23:27.704006 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:27.703982 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" event={"ID":"c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb","Type":"ContainerStarted","Data":"9c0e5d8bb4c8ccfe757841330f91db9150e4edfa282a57fade5f9ac2f7a3e10e"} Sep 11 14:23:27.704118 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:27.704039 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" Sep 11 14:23:27.754252 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:27.754162 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" podStartSLOduration=1.7460136830000001 podStartE2EDuration="4.754143146s" totalImagesPullingTime="3.008129463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:27.750498061 +0000 UTC m=+495.835954425" watchObservedRunningTime="2026-09-11 14:23:27.754143146 +0000 UTC m=+495.839599508" Sep 11 14:23:28.045500 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:28.045411 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr"] Sep 11 14:23:28.049037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:28.049015 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:28.052188 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:28.052171 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jgdfq\"" Sep 11 14:23:28.071984 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:28.071961 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr"] Sep 11 14:23:28.119607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:28.119579 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxrv2\" (UniqueName: \"kubernetes.io/projected/5ea79808-0ab0-44c3-a7c0-68f40bba9575-kube-api-access-wxrv2\") pod \"limitador-operator-controller-manager-74495c69f8-4rnrr\" (UID: \"5ea79808-0ab0-44c3-a7c0-68f40bba9575\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:28.220408 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:28.220366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxrv2\" (UniqueName: \"kubernetes.io/projected/5ea79808-0ab0-44c3-a7c0-68f40bba9575-kube-api-access-wxrv2\") pod \"limitador-operator-controller-manager-74495c69f8-4rnrr\" (UID: \"5ea79808-0ab0-44c3-a7c0-68f40bba9575\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:28.231449 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:28.231417 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxrv2\" (UniqueName: \"kubernetes.io/projected/5ea79808-0ab0-44c3-a7c0-68f40bba9575-kube-api-access-wxrv2\") pod \"limitador-operator-controller-manager-74495c69f8-4rnrr\" (UID: \"5ea79808-0ab0-44c3-a7c0-68f40bba9575\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:28.359877 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:28.359831 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:28.672069 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:28.672037 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr"] Sep 11 14:23:28.674307 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:23:28.674274 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea79808_0ab0_44c3_a7c0_68f40bba9575.slice/crio-c3cf9afce8057a114e48fc95d2152e1fe93a0d1ddf72deb9d2decaeb68eb3f29 WatchSource:0}: Error finding container c3cf9afce8057a114e48fc95d2152e1fe93a0d1ddf72deb9d2decaeb68eb3f29: Status 404 returned error can't find the container with id c3cf9afce8057a114e48fc95d2152e1fe93a0d1ddf72deb9d2decaeb68eb3f29 Sep 11 14:23:28.710070 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:28.710038 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" event={"ID":"5ea79808-0ab0-44c3-a7c0-68f40bba9575","Type":"ContainerStarted","Data":"c3cf9afce8057a114e48fc95d2152e1fe93a0d1ddf72deb9d2decaeb68eb3f29"} Sep 11 14:23:29.719464 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:29.719426 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" event={"ID":"23e76c7f-aeae-4756-bb38-eba9de65e417","Type":"ContainerStarted","Data":"6cdbac991650f3f80858770648a8b1f2a3eccd2ce12c37049f04137523eb067c"} Sep 11 14:23:29.719956 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:29.719620 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" Sep 11 14:23:29.752750 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:29.751842 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" podStartSLOduration=1.616129172 podStartE2EDuration="3.751821178s" totalImagesPullingTime="2.135692006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:29.751225363 +0000 UTC m=+497.836681727" watchObservedRunningTime="2026-09-11 14:23:29.751821178 +0000 UTC m=+497.837277539" Sep 11 14:23:30.726256 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:30.726173 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" event={"ID":"5ea79808-0ab0-44c3-a7c0-68f40bba9575","Type":"ContainerStarted","Data":"70c73b86e300afcc4fcaea7834cd3be912852764035478a5c3529b49cf1d0aa5"} Sep 11 14:23:30.726602 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:30.726449 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:30.759681 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:30.759638 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" podStartSLOduration=1.018118701 podStartE2EDuration="2.759623109s" totalImagesPullingTime="1.741504408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:30.758707093 +0000 UTC m=+498.844163453" watchObservedRunningTime="2026-09-11 14:23:30.759623109 +0000 UTC m=+498.845079468" Sep 11 14:23:32.991321 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:32.991285 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn"] Sep 11 14:23:32.994640 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:32.994624 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:32.998451 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:32.998429 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kvhdm\"" Sep 11 14:23:33.040294 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:33.040264 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn"] Sep 11 14:23:33.063374 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:33.063343 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcad0846-9c72-4fe2-9877-6931dcb2cf95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" (UID: \"fcad0846-9c72-4fe2-9877-6931dcb2cf95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:33.063505 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:33.063379 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwm9v\" (UniqueName: \"kubernetes.io/projected/fcad0846-9c72-4fe2-9877-6931dcb2cf95-kube-api-access-hwm9v\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" (UID: \"fcad0846-9c72-4fe2-9877-6931dcb2cf95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:33.164720 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:33.164690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcad0846-9c72-4fe2-9877-6931dcb2cf95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" (UID: \"fcad0846-9c72-4fe2-9877-6931dcb2cf95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:33.164720 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:33.164722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwm9v\" (UniqueName: \"kubernetes.io/projected/fcad0846-9c72-4fe2-9877-6931dcb2cf95-kube-api-access-hwm9v\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" (UID: \"fcad0846-9c72-4fe2-9877-6931dcb2cf95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:33.165118 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:33.165095 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcad0846-9c72-4fe2-9877-6931dcb2cf95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" (UID: \"fcad0846-9c72-4fe2-9877-6931dcb2cf95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:33.191473 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:33.191449 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwm9v\" (UniqueName: \"kubernetes.io/projected/fcad0846-9c72-4fe2-9877-6931dcb2cf95-kube-api-access-hwm9v\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" (UID: \"fcad0846-9c72-4fe2-9877-6931dcb2cf95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:33.304700 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:33.304628 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:33.596203 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:33.596128 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn"] Sep 11 14:23:33.599344 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:23:33.599303 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcad0846_9c72_4fe2_9877_6931dcb2cf95.slice/crio-23f2e127824446b412fa8b6242c26027940da8f7aff0b6ee67fbba0b5d1cb0c3 WatchSource:0}: Error finding container 23f2e127824446b412fa8b6242c26027940da8f7aff0b6ee67fbba0b5d1cb0c3: Status 404 returned error can't find the container with id 23f2e127824446b412fa8b6242c26027940da8f7aff0b6ee67fbba0b5d1cb0c3 Sep 11 14:23:33.743118 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:33.743088 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" event={"ID":"fcad0846-9c72-4fe2-9877-6931dcb2cf95","Type":"ContainerStarted","Data":"23f2e127824446b412fa8b6242c26027940da8f7aff0b6ee67fbba0b5d1cb0c3"} Sep 11 14:23:38.713070 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:38.713036 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" Sep 11 14:23:40.729207 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:40.729178 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" Sep 11 14:23:41.733335 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:41.733306 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:43.804637 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:43.804597 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" event={"ID":"fcad0846-9c72-4fe2-9877-6931dcb2cf95","Type":"ContainerStarted","Data":"fe17b2cfb26976ef9921b99f33ffc1893248772e2bf48cb7e0b18fe0c8e1bda2"} Sep 11 14:23:43.805226 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:43.805201 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:43.866922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:43.866871 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" podStartSLOduration=2.165088244 podStartE2EDuration="11.866853831s" totalImagesPullingTime="9.701765587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:43.863949336 +0000 UTC m=+511.949405696" watchObservedRunningTime="2026-09-11 14:23:43.866853831 +0000 UTC m=+511.952310190" Sep 11 14:23:45.984866 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:45.984833 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6"] Sep 11 14:23:45.988419 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:45.988399 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" Sep 11 14:23:45.991374 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:45.991354 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ftgll\"" Sep 11 14:23:45.991789 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:45.991767 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 11 14:23:45.991941 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:45.991924 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 11 14:23:46.002486 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:46.002465 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6"] Sep 11 14:23:46.065263 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:46.065235 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-96jj6\" (UID: \"f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" Sep 11 14:23:46.065394 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:46.065293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcbzq\" (UniqueName: \"kubernetes.io/projected/f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76-kube-api-access-tcbzq\") pod \"kuadrant-console-plugin-fbdd4f445-96jj6\" (UID: \"f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" Sep 11 14:23:46.065440 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:46.065393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-96jj6\" (UID: \"f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" Sep 11 14:23:46.166150 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:46.166121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcbzq\" (UniqueName: \"kubernetes.io/projected/f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76-kube-api-access-tcbzq\") pod \"kuadrant-console-plugin-fbdd4f445-96jj6\" (UID: \"f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" Sep 11 14:23:46.166295 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:46.166171 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-96jj6\" (UID: \"f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" Sep 11 14:23:46.166295 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:46.166194 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-96jj6\" (UID: \"f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" Sep 11 14:23:46.166794 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:46.166769 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-96jj6\" (UID: \"f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" Sep 11 14:23:46.168495 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:46.168477 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-96jj6\" (UID: \"f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" Sep 11 14:23:46.179306 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:46.179282 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcbzq\" (UniqueName: \"kubernetes.io/projected/f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76-kube-api-access-tcbzq\") pod \"kuadrant-console-plugin-fbdd4f445-96jj6\" (UID: \"f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" Sep 11 14:23:46.297395 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:46.297327 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" Sep 11 14:23:46.583546 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:46.583522 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6"] Sep 11 14:23:46.584239 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:23:46.584214 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9b7ccb0_ec01_49ec_a1a5_abd48ce45f76.slice/crio-1b330150462a39ad7213d8124758489315d037f7c36b55d4627d3e92c674c8c5 WatchSource:0}: Error finding container 1b330150462a39ad7213d8124758489315d037f7c36b55d4627d3e92c674c8c5: Status 404 returned error can't find the container with id 1b330150462a39ad7213d8124758489315d037f7c36b55d4627d3e92c674c8c5 Sep 11 14:23:46.821554 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:46.821516 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" event={"ID":"f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76","Type":"ContainerStarted","Data":"1b330150462a39ad7213d8124758489315d037f7c36b55d4627d3e92c674c8c5"} Sep 11 14:23:51.851951 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:51.851915 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" event={"ID":"f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76","Type":"ContainerStarted","Data":"d3aecc64088d99904310d5da7a88761126c8e3f2570f1827629eefa65cb7f600"} Sep 11 14:23:51.887523 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:51.887477 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-96jj6" podStartSLOduration=2.416199512 podStartE2EDuration="6.887461952s" totalImagesPullingTime="4.47126244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:51.887101721 +0000 UTC m=+519.972558082" watchObservedRunningTime="2026-09-11 14:23:51.887461952 +0000 UTC m=+519.972918311" Sep 11 14:23:54.811604 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:54.811572 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:56.722690 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:56.722658 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54"] Sep 11 14:23:56.727368 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:56.727343 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" Sep 11 14:23:56.751180 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:56.751158 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54"] Sep 11 14:23:56.859189 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:56.859167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cthj8\" (UniqueName: \"kubernetes.io/projected/699da8bb-92e3-4fd6-af90-c65d26a5e38d-kube-api-access-cthj8\") pod \"kuadrant-operator-controller-manager-5f685c496f-45g54\" (UID: \"699da8bb-92e3-4fd6-af90-c65d26a5e38d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" Sep 11 14:23:56.859339 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:56.859212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/699da8bb-92e3-4fd6-af90-c65d26a5e38d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-45g54\" (UID: \"699da8bb-92e3-4fd6-af90-c65d26a5e38d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" Sep 11 14:23:56.878364 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:56.878341 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54"] Sep 11 14:23:56.878578 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:56.878559 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-cthj8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" podUID="699da8bb-92e3-4fd6-af90-c65d26a5e38d" Sep 11 14:23:56.960259 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:56.960236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cthj8\" (UniqueName: \"kubernetes.io/projected/699da8bb-92e3-4fd6-af90-c65d26a5e38d-kube-api-access-cthj8\") pod \"kuadrant-operator-controller-manager-5f685c496f-45g54\" (UID: \"699da8bb-92e3-4fd6-af90-c65d26a5e38d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" Sep 11 14:23:56.960364 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:56.960275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/699da8bb-92e3-4fd6-af90-c65d26a5e38d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-45g54\" (UID: \"699da8bb-92e3-4fd6-af90-c65d26a5e38d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" Sep 11 14:23:56.960587 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:56.960570 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/699da8bb-92e3-4fd6-af90-c65d26a5e38d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-45g54\" (UID: \"699da8bb-92e3-4fd6-af90-c65d26a5e38d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" Sep 11 14:23:56.979353 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:56.979285 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cthj8\" (UniqueName: \"kubernetes.io/projected/699da8bb-92e3-4fd6-af90-c65d26a5e38d-kube-api-access-cthj8\") pod \"kuadrant-operator-controller-manager-5f685c496f-45g54\" (UID: \"699da8bb-92e3-4fd6-af90-c65d26a5e38d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" Sep 11 14:23:57.014370 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.014343 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd"] Sep 11 14:23:57.017933 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.017917 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:57.106931 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.106904 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd"] Sep 11 14:23:57.161958 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.161925 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/880692f5-a7a6-428c-b9be-bb543b5684ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-824qd\" (UID: \"880692f5-a7a6-428c-b9be-bb543b5684ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:57.162107 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.161975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knsx2\" (UniqueName: \"kubernetes.io/projected/880692f5-a7a6-428c-b9be-bb543b5684ca-kube-api-access-knsx2\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-824qd\" (UID: \"880692f5-a7a6-428c-b9be-bb543b5684ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:57.262798 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.262686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/880692f5-a7a6-428c-b9be-bb543b5684ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-824qd\" (UID: \"880692f5-a7a6-428c-b9be-bb543b5684ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:57.262798 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.262760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsx2\" (UniqueName: \"kubernetes.io/projected/880692f5-a7a6-428c-b9be-bb543b5684ca-kube-api-access-knsx2\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-824qd\" (UID: \"880692f5-a7a6-428c-b9be-bb543b5684ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:57.263054 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.263034 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/880692f5-a7a6-428c-b9be-bb543b5684ca-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-824qd\" (UID: \"880692f5-a7a6-428c-b9be-bb543b5684ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:57.279623 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.279601 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knsx2\" (UniqueName: \"kubernetes.io/projected/880692f5-a7a6-428c-b9be-bb543b5684ca-kube-api-access-knsx2\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-824qd\" (UID: \"880692f5-a7a6-428c-b9be-bb543b5684ca\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:57.327571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.327549 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:57.458187 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.458159 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54"] Sep 11 14:23:57.467025 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.467001 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54"] Sep 11 14:23:57.481076 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.481051 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn"] Sep 11 14:23:57.481316 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.481267 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" containerName="manager" containerID="cri-o://fe17b2cfb26976ef9921b99f33ffc1893248772e2bf48cb7e0b18fe0c8e1bda2" gracePeriod=2 Sep 11 14:23:57.485095 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.485033 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:57.486222 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.486195 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn"] Sep 11 14:23:57.510591 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.510564 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd"] Sep 11 14:23:57.522506 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.522426 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq"] Sep 11 14:23:57.522872 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.522854 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" containerName="manager" Sep 11 14:23:57.522872 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.522871 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" containerName="manager" Sep 11 14:23:57.523027 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.522957 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" containerName="manager" Sep 11 14:23:57.526202 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.526166 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd"] Sep 11 14:23:57.526341 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.526283 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" Sep 11 14:23:57.551629 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.551605 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n"] Sep 11 14:23:57.558243 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.558213 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr"] Sep 11 14:23:57.558447 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.558325 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:23:57.558959 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.558930 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" podUID="5ea79808-0ab0-44c3-a7c0-68f40bba9575" containerName="manager" containerID="cri-o://70c73b86e300afcc4fcaea7834cd3be912852764035478a5c3529b49cf1d0aa5" gracePeriod=2 Sep 11 14:23:57.560249 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.560219 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq"] Sep 11 14:23:57.567870 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.567802 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr"] Sep 11 14:23:57.582358 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.582336 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n"] Sep 11 14:23:57.593359 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.593325 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:57.596911 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.596875 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4rnrr\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="5ea79808-0ab0-44c3-a7c0-68f40bba9575" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:57.601101 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.600260 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5k6br"] Sep 11 14:23:57.607133 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.601242 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea79808-0ab0-44c3-a7c0-68f40bba9575" containerName="manager" Sep 11 14:23:57.607133 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.601262 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea79808-0ab0-44c3-a7c0-68f40bba9575" containerName="manager" Sep 11 14:23:57.607133 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.601482 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ea79808-0ab0-44c3-a7c0-68f40bba9575" containerName="manager" Sep 11 14:23:57.607348 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.607092 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:57.610777 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.610755 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5k6br" Sep 11 14:23:57.622594 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.622574 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5k6br"] Sep 11 14:23:57.645290 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.645250 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:57.647471 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.647442 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4rnrr\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="5ea79808-0ab0-44c3-a7c0-68f40bba9575" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:57.666104 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.666083 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6nk\" (UniqueName: \"kubernetes.io/projected/bc3863fd-b810-45a6-a97c-a5144ba4eb98-kube-api-access-pg6nk\") pod \"limitador-operator-controller-manager-74495c69f8-5k6br\" (UID: \"bc3863fd-b810-45a6-a97c-a5144ba4eb98\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5k6br" Sep 11 14:23:57.666203 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.666121 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d616a45a-24e4-455f-a5b5-b487c7a4708d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-jhr6n\" (UID: \"d616a45a-24e4-455f-a5b5-b487c7a4708d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:23:57.666203 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.666164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2gqq\" (UniqueName: \"kubernetes.io/projected/d616a45a-24e4-455f-a5b5-b487c7a4708d-kube-api-access-h2gqq\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-jhr6n\" (UID: \"d616a45a-24e4-455f-a5b5-b487c7a4708d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:23:57.666203 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.666191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9txdl\" (UniqueName: \"kubernetes.io/projected/b0e528e6-f8a9-4511-9a80-b68f6de1e83b-kube-api-access-9txdl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vqcvq\" (UID: \"b0e528e6-f8a9-4511-9a80-b68f6de1e83b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" Sep 11 14:23:57.666367 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.666235 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0e528e6-f8a9-4511-9a80-b68f6de1e83b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vqcvq\" (UID: \"b0e528e6-f8a9-4511-9a80-b68f6de1e83b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" Sep 11 14:23:57.736386 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.736344 2578 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 11 14:23:57.736386 ip-10-0-131-57 kubenswrapper[2578]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-824qd_kuadrant-system_880692f5-a7a6-428c-b9be-bb543b5684ca_0(9311e3f5d2f17a918033f6d287835e5539241586197f4568fe987d0b37597771): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-824qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"9311e3f5d2f17a918033f6d287835e5539241586197f4568fe987d0b37597771" Netns:"/var/run/netns/89c77a66-1a0f-4b2f-b84e-e092faf02b2c" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-824qd;K8S_POD_INFRA_CONTAINER_ID=9311e3f5d2f17a918033f6d287835e5539241586197f4568fe987d0b37597771;K8S_POD_UID=880692f5-a7a6-428c-b9be-bb543b5684ca" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd/880692f5-a7a6-428c-b9be-bb543b5684ca:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd 9311e3f5d2f17a918033f6d287835e5539241586197f4568fe987d0b37597771 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:2c [10.134.0.44/23] Sep 11 14:23:57.736386 ip-10-0-131-57 kubenswrapper[2578]: ' Sep 11 14:23:57.736386 ip-10-0-131-57 kubenswrapper[2578]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 11 14:23:57.736386 ip-10-0-131-57 kubenswrapper[2578]: > Sep 11 14:23:57.736859 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.736424 2578 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 11 14:23:57.736859 ip-10-0-131-57 kubenswrapper[2578]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-824qd_kuadrant-system_880692f5-a7a6-428c-b9be-bb543b5684ca_0(9311e3f5d2f17a918033f6d287835e5539241586197f4568fe987d0b37597771): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-824qd to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"9311e3f5d2f17a918033f6d287835e5539241586197f4568fe987d0b37597771" Netns:"/var/run/netns/89c77a66-1a0f-4b2f-b84e-e092faf02b2c" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-824qd;K8S_POD_INFRA_CONTAINER_ID=9311e3f5d2f17a918033f6d287835e5539241586197f4568fe987d0b37597771;K8S_POD_UID=880692f5-a7a6-428c-b9be-bb543b5684ca" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd/880692f5-a7a6-428c-b9be-bb543b5684ca:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd 9311e3f5d2f17a918033f6d287835e5539241586197f4568fe987d0b37597771 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:2c [10.134.0.44/23] Sep 11 14:23:57.736859 ip-10-0-131-57 kubenswrapper[2578]: ' Sep 11 14:23:57.736859 ip-10-0-131-57 kubenswrapper[2578]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 11 14:23:57.736859 ip-10-0-131-57 kubenswrapper[2578]: > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:57.766752 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.766694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6nk\" (UniqueName: \"kubernetes.io/projected/bc3863fd-b810-45a6-a97c-a5144ba4eb98-kube-api-access-pg6nk\") pod \"limitador-operator-controller-manager-74495c69f8-5k6br\" (UID: \"bc3863fd-b810-45a6-a97c-a5144ba4eb98\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5k6br" Sep 11 14:23:57.766952 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.766768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d616a45a-24e4-455f-a5b5-b487c7a4708d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-jhr6n\" (UID: \"d616a45a-24e4-455f-a5b5-b487c7a4708d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:23:57.766952 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.766801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2gqq\" (UniqueName: \"kubernetes.io/projected/d616a45a-24e4-455f-a5b5-b487c7a4708d-kube-api-access-h2gqq\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-jhr6n\" (UID: \"d616a45a-24e4-455f-a5b5-b487c7a4708d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:23:57.766952 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.766829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9txdl\" (UniqueName: \"kubernetes.io/projected/b0e528e6-f8a9-4511-9a80-b68f6de1e83b-kube-api-access-9txdl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vqcvq\" (UID: \"b0e528e6-f8a9-4511-9a80-b68f6de1e83b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" Sep 11 14:23:57.766952 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.766876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0e528e6-f8a9-4511-9a80-b68f6de1e83b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vqcvq\" (UID: \"b0e528e6-f8a9-4511-9a80-b68f6de1e83b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" Sep 11 14:23:57.767412 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.767204 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d616a45a-24e4-455f-a5b5-b487c7a4708d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-jhr6n\" (UID: \"d616a45a-24e4-455f-a5b5-b487c7a4708d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:23:57.767412 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.767246 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0e528e6-f8a9-4511-9a80-b68f6de1e83b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vqcvq\" (UID: \"b0e528e6-f8a9-4511-9a80-b68f6de1e83b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" Sep 11 14:23:57.780368 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.780348 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2gqq\" (UniqueName: \"kubernetes.io/projected/d616a45a-24e4-455f-a5b5-b487c7a4708d-kube-api-access-h2gqq\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-jhr6n\" (UID: \"d616a45a-24e4-455f-a5b5-b487c7a4708d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:23:57.780651 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.780628 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6nk\" (UniqueName: \"kubernetes.io/projected/bc3863fd-b810-45a6-a97c-a5144ba4eb98-kube-api-access-pg6nk\") pod \"limitador-operator-controller-manager-74495c69f8-5k6br\" (UID: \"bc3863fd-b810-45a6-a97c-a5144ba4eb98\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5k6br" Sep 11 14:23:57.781148 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.781128 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9txdl\" (UniqueName: \"kubernetes.io/projected/b0e528e6-f8a9-4511-9a80-b68f6de1e83b-kube-api-access-9txdl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-vqcvq\" (UID: \"b0e528e6-f8a9-4511-9a80-b68f6de1e83b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" Sep 11 14:23:57.787211 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.787191 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:57.789959 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.789934 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4rnrr\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="5ea79808-0ab0-44c3-a7c0-68f40bba9575" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:57.793473 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.793451 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:57.795018 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.795003 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:57.797712 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.797690 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4rnrr\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="5ea79808-0ab0-44c3-a7c0-68f40bba9575" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:57.800849 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.800825 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:57.858630 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.858605 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" Sep 11 14:23:57.867444 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.867422 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwm9v\" (UniqueName: \"kubernetes.io/projected/fcad0846-9c72-4fe2-9877-6931dcb2cf95-kube-api-access-hwm9v\") pod \"fcad0846-9c72-4fe2-9877-6931dcb2cf95\" (UID: \"fcad0846-9c72-4fe2-9877-6931dcb2cf95\") " Sep 11 14:23:57.867540 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.867501 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcad0846-9c72-4fe2-9877-6931dcb2cf95-extensions-socket-volume\") pod \"fcad0846-9c72-4fe2-9877-6931dcb2cf95\" (UID: \"fcad0846-9c72-4fe2-9877-6931dcb2cf95\") " Sep 11 14:23:57.867601 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.867542 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxrv2\" (UniqueName: \"kubernetes.io/projected/5ea79808-0ab0-44c3-a7c0-68f40bba9575-kube-api-access-wxrv2\") pod \"5ea79808-0ab0-44c3-a7c0-68f40bba9575\" (UID: \"5ea79808-0ab0-44c3-a7c0-68f40bba9575\") " Sep 11 14:23:57.868016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.867994 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcad0846-9c72-4fe2-9877-6931dcb2cf95-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fcad0846-9c72-4fe2-9877-6931dcb2cf95" (UID: "fcad0846-9c72-4fe2-9877-6931dcb2cf95"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:57.869703 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.869685 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcad0846-9c72-4fe2-9877-6931dcb2cf95-kube-api-access-hwm9v" (OuterVolumeSpecName: "kube-api-access-hwm9v") pod "fcad0846-9c72-4fe2-9877-6931dcb2cf95" (UID: "fcad0846-9c72-4fe2-9877-6931dcb2cf95"). InnerVolumeSpecName "kube-api-access-hwm9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:57.869703 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.869707 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea79808-0ab0-44c3-a7c0-68f40bba9575-kube-api-access-wxrv2" (OuterVolumeSpecName: "kube-api-access-wxrv2") pod "5ea79808-0ab0-44c3-a7c0-68f40bba9575" (UID: "5ea79808-0ab0-44c3-a7c0-68f40bba9575"). InnerVolumeSpecName "kube-api-access-wxrv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:57.872851 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.872829 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:23:57.888163 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.888132 2578 generic.go:356] "Generic (PLEG): container finished" podID="5ea79808-0ab0-44c3-a7c0-68f40bba9575" containerID="70c73b86e300afcc4fcaea7834cd3be912852764035478a5c3529b49cf1d0aa5" exitCode=0 Sep 11 14:23:57.888265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.888185 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:57.888331 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.888317 2578 scope.go:117] "RemoveContainer" containerID="70c73b86e300afcc4fcaea7834cd3be912852764035478a5c3529b49cf1d0aa5" Sep 11 14:23:57.890231 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.890207 2578 generic.go:356] "Generic (PLEG): container finished" podID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" containerID="fe17b2cfb26976ef9921b99f33ffc1893248772e2bf48cb7e0b18fe0c8e1bda2" exitCode=0 Sep 11 14:23:57.890326 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.890312 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" Sep 11 14:23:57.890326 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.890320 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:57.890473 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.890433 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:57.891767 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.891706 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4rnrr\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="5ea79808-0ab0-44c3-a7c0-68f40bba9575" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:57.894690 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.894513 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:57.898310 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.898267 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4rnrr\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="5ea79808-0ab0-44c3-a7c0-68f40bba9575" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:57.901349 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.901320 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-824qd\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="880692f5-a7a6-428c-b9be-bb543b5684ca" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:57.901916 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.901867 2578 scope.go:117] "RemoveContainer" containerID="70c73b86e300afcc4fcaea7834cd3be912852764035478a5c3529b49cf1d0aa5" Sep 11 14:23:57.902309 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.902190 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c73b86e300afcc4fcaea7834cd3be912852764035478a5c3529b49cf1d0aa5\": container with ID starting with 70c73b86e300afcc4fcaea7834cd3be912852764035478a5c3529b49cf1d0aa5 not found: ID does not exist" containerID="70c73b86e300afcc4fcaea7834cd3be912852764035478a5c3529b49cf1d0aa5" Sep 11 14:23:57.902309 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.902225 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c73b86e300afcc4fcaea7834cd3be912852764035478a5c3529b49cf1d0aa5"} err="failed to get container status \"70c73b86e300afcc4fcaea7834cd3be912852764035478a5c3529b49cf1d0aa5\": rpc error: code = NotFound desc = could not find container \"70c73b86e300afcc4fcaea7834cd3be912852764035478a5c3529b49cf1d0aa5\": container with ID starting with 70c73b86e300afcc4fcaea7834cd3be912852764035478a5c3529b49cf1d0aa5 not found: ID does not exist" Sep 11 14:23:57.902309 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.902250 2578 scope.go:117] "RemoveContainer" containerID="fe17b2cfb26976ef9921b99f33ffc1893248772e2bf48cb7e0b18fe0c8e1bda2" Sep 11 14:23:57.902902 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.902809 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:57.904085 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.904065 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-45g54\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="699da8bb-92e3-4fd6-af90-c65d26a5e38d" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" Sep 11 14:23:57.906906 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.906864 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" Sep 11 14:23:57.908083 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.908041 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:57.911199 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.911174 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4rnrr\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="5ea79808-0ab0-44c3-a7c0-68f40bba9575" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4rnrr" Sep 11 14:23:57.914522 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.914494 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-824qd\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="880692f5-a7a6-428c-b9be-bb543b5684ca" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:57.916605 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.916542 2578 scope.go:117] "RemoveContainer" containerID="fe17b2cfb26976ef9921b99f33ffc1893248772e2bf48cb7e0b18fe0c8e1bda2" Sep 11 14:23:57.916996 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.916970 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe17b2cfb26976ef9921b99f33ffc1893248772e2bf48cb7e0b18fe0c8e1bda2\": container with ID starting with fe17b2cfb26976ef9921b99f33ffc1893248772e2bf48cb7e0b18fe0c8e1bda2 not found: ID does not exist" containerID="fe17b2cfb26976ef9921b99f33ffc1893248772e2bf48cb7e0b18fe0c8e1bda2" Sep 11 14:23:57.917135 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.917012 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe17b2cfb26976ef9921b99f33ffc1893248772e2bf48cb7e0b18fe0c8e1bda2"} err="failed to get container status \"fe17b2cfb26976ef9921b99f33ffc1893248772e2bf48cb7e0b18fe0c8e1bda2\": rpc error: code = NotFound desc = could not find container \"fe17b2cfb26976ef9921b99f33ffc1893248772e2bf48cb7e0b18fe0c8e1bda2\": container with ID starting with fe17b2cfb26976ef9921b99f33ffc1893248772e2bf48cb7e0b18fe0c8e1bda2 not found: ID does not exist" Sep 11 14:23:57.917226 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.916980 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-45g54\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="699da8bb-92e3-4fd6-af90-c65d26a5e38d" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" Sep 11 14:23:57.920400 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:57.920200 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-8kkpn\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8kkpn" Sep 11 14:23:57.923053 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.922902 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5k6br" Sep 11 14:23:57.968580 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.968173 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/699da8bb-92e3-4fd6-af90-c65d26a5e38d-extensions-socket-volume\") pod \"699da8bb-92e3-4fd6-af90-c65d26a5e38d\" (UID: \"699da8bb-92e3-4fd6-af90-c65d26a5e38d\") " Sep 11 14:23:57.968580 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.968213 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cthj8\" (UniqueName: \"kubernetes.io/projected/699da8bb-92e3-4fd6-af90-c65d26a5e38d-kube-api-access-cthj8\") pod \"699da8bb-92e3-4fd6-af90-c65d26a5e38d\" (UID: \"699da8bb-92e3-4fd6-af90-c65d26a5e38d\") " Sep 11 14:23:57.968580 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.968260 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knsx2\" (UniqueName: \"kubernetes.io/projected/880692f5-a7a6-428c-b9be-bb543b5684ca-kube-api-access-knsx2\") pod \"880692f5-a7a6-428c-b9be-bb543b5684ca\" (UID: \"880692f5-a7a6-428c-b9be-bb543b5684ca\") " Sep 11 14:23:57.968580 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.968292 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/880692f5-a7a6-428c-b9be-bb543b5684ca-extensions-socket-volume\") pod \"880692f5-a7a6-428c-b9be-bb543b5684ca\" (UID: \"880692f5-a7a6-428c-b9be-bb543b5684ca\") " Sep 11 14:23:57.968580 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.968420 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcad0846-9c72-4fe2-9877-6931dcb2cf95-extensions-socket-volume\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:57.968580 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.968430 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxrv2\" (UniqueName: \"kubernetes.io/projected/5ea79808-0ab0-44c3-a7c0-68f40bba9575-kube-api-access-wxrv2\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:57.968580 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.968439 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwm9v\" (UniqueName: \"kubernetes.io/projected/fcad0846-9c72-4fe2-9877-6931dcb2cf95-kube-api-access-hwm9v\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:57.968580 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.968510 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699da8bb-92e3-4fd6-af90-c65d26a5e38d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "699da8bb-92e3-4fd6-af90-c65d26a5e38d" (UID: "699da8bb-92e3-4fd6-af90-c65d26a5e38d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:57.969096 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.968805 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880692f5-a7a6-428c-b9be-bb543b5684ca-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "880692f5-a7a6-428c-b9be-bb543b5684ca" (UID: "880692f5-a7a6-428c-b9be-bb543b5684ca"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:57.970920 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.970884 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699da8bb-92e3-4fd6-af90-c65d26a5e38d-kube-api-access-cthj8" (OuterVolumeSpecName: "kube-api-access-cthj8") pod "699da8bb-92e3-4fd6-af90-c65d26a5e38d" (UID: "699da8bb-92e3-4fd6-af90-c65d26a5e38d"). InnerVolumeSpecName "kube-api-access-cthj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:57.971127 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:57.971008 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880692f5-a7a6-428c-b9be-bb543b5684ca-kube-api-access-knsx2" (OuterVolumeSpecName: "kube-api-access-knsx2") pod "880692f5-a7a6-428c-b9be-bb543b5684ca" (UID: "880692f5-a7a6-428c-b9be-bb543b5684ca"). InnerVolumeSpecName "kube-api-access-knsx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:58.069131 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.069061 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/699da8bb-92e3-4fd6-af90-c65d26a5e38d-extensions-socket-volume\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:58.069131 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.069085 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cthj8\" (UniqueName: \"kubernetes.io/projected/699da8bb-92e3-4fd6-af90-c65d26a5e38d-kube-api-access-cthj8\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:58.069131 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.069095 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knsx2\" (UniqueName: \"kubernetes.io/projected/880692f5-a7a6-428c-b9be-bb543b5684ca-kube-api-access-knsx2\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:58.069131 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.069104 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/880692f5-a7a6-428c-b9be-bb543b5684ca-extensions-socket-volume\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:23:58.201041 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:23:58.201015 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e528e6_f8a9_4511_9a80_b68f6de1e83b.slice/crio-213e18c1a884fe33f633114b1cc650ce26e34b84ee1e53c34f7d10aff4f3b13e WatchSource:0}: Error finding container 213e18c1a884fe33f633114b1cc650ce26e34b84ee1e53c34f7d10aff4f3b13e: Status 404 returned error can't find the container with id 213e18c1a884fe33f633114b1cc650ce26e34b84ee1e53c34f7d10aff4f3b13e Sep 11 14:23:58.201965 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.201936 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq"] Sep 11 14:23:58.202122 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:23:58.202098 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd616a45a_24e4_455f_a5b5_b487c7a4708d.slice/crio-0758eccfb4cc89543f2a61cb481b1807d69518cd8e01908d7c5e66b2259ea485 WatchSource:0}: Error finding container 0758eccfb4cc89543f2a61cb481b1807d69518cd8e01908d7c5e66b2259ea485: Status 404 returned error can't find the container with id 0758eccfb4cc89543f2a61cb481b1807d69518cd8e01908d7c5e66b2259ea485 Sep 11 14:23:58.204698 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.204654 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n"] Sep 11 14:23:58.286860 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.286840 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5k6br"] Sep 11 14:23:58.287607 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:23:58.287588 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3863fd_b810_45a6_a97c_a5144ba4eb98.slice/crio-84f26bdde4fa4b5ed40bb42cf3737d07830c68188e55ebf3fd25961b1888cf93 WatchSource:0}: Error finding container 84f26bdde4fa4b5ed40bb42cf3737d07830c68188e55ebf3fd25961b1888cf93: Status 404 returned error can't find the container with id 84f26bdde4fa4b5ed40bb42cf3737d07830c68188e55ebf3fd25961b1888cf93 Sep 11 14:23:58.464781 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.464749 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ea79808-0ab0-44c3-a7c0-68f40bba9575" path="/var/lib/kubelet/pods/5ea79808-0ab0-44c3-a7c0-68f40bba9575/volumes" Sep 11 14:23:58.465155 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.465140 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="699da8bb-92e3-4fd6-af90-c65d26a5e38d" path="/var/lib/kubelet/pods/699da8bb-92e3-4fd6-af90-c65d26a5e38d/volumes" Sep 11 14:23:58.465369 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.465358 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="880692f5-a7a6-428c-b9be-bb543b5684ca" path="/var/lib/kubelet/pods/880692f5-a7a6-428c-b9be-bb543b5684ca/volumes" Sep 11 14:23:58.465634 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.465619 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcad0846-9c72-4fe2-9877-6931dcb2cf95" path="/var/lib/kubelet/pods/fcad0846-9c72-4fe2-9877-6931dcb2cf95/volumes" Sep 11 14:23:58.897798 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.897761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" event={"ID":"b0e528e6-f8a9-4511-9a80-b68f6de1e83b","Type":"ContainerStarted","Data":"43eeb5f2daa7a24a1c2ea96cbbef013f893c3353e5735b5c59b591fb271f01b8"} Sep 11 14:23:58.898224 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.897935 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" Sep 11 14:23:58.898224 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.897985 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" event={"ID":"b0e528e6-f8a9-4511-9a80-b68f6de1e83b","Type":"ContainerStarted","Data":"213e18c1a884fe33f633114b1cc650ce26e34b84ee1e53c34f7d10aff4f3b13e"} Sep 11 14:23:58.900302 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.900279 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5k6br" event={"ID":"bc3863fd-b810-45a6-a97c-a5144ba4eb98","Type":"ContainerStarted","Data":"3b1f761931f7c9548e95d4b256191ccf12bf117b32b7b1287ab97d1544b71ffe"} Sep 11 14:23:58.900433 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.900321 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5k6br" event={"ID":"bc3863fd-b810-45a6-a97c-a5144ba4eb98","Type":"ContainerStarted","Data":"84f26bdde4fa4b5ed40bb42cf3737d07830c68188e55ebf3fd25961b1888cf93"} Sep 11 14:23:58.900433 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.900347 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5k6br" Sep 11 14:23:58.901935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.901914 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" Sep 11 14:23:58.902046 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.901942 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" event={"ID":"d616a45a-24e4-455f-a5b5-b487c7a4708d","Type":"ContainerStarted","Data":"9e657f504347fb52a456b1aabd78fe932ee8b0fd1794daffdd119fcb6ce3c503"} Sep 11 14:23:58.902046 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.901968 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" event={"ID":"d616a45a-24e4-455f-a5b5-b487c7a4708d","Type":"ContainerStarted","Data":"0758eccfb4cc89543f2a61cb481b1807d69518cd8e01908d7c5e66b2259ea485"} Sep 11 14:23:58.902046 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.901987 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:23:58.902046 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.901969 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:23:58.970572 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:58.970527 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" podStartSLOduration=1.970513784 podStartE2EDuration="1.970513784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:58.968472869 +0000 UTC m=+527.053929230" watchObservedRunningTime="2026-09-11 14:23:58.970513784 +0000 UTC m=+527.055970143" Sep 11 14:23:58.971427 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:58.971405 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-45g54\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="699da8bb-92e3-4fd6-af90-c65d26a5e38d" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-45g54" Sep 11 14:23:59.001025 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:59.000979 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5k6br" podStartSLOduration=2.000964922 podStartE2EDuration="2.000964922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:59.000357033 +0000 UTC m=+527.085813422" watchObservedRunningTime="2026-09-11 14:23:59.000964922 +0000 UTC m=+527.086421281" Sep 11 14:23:59.039190 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:23:59.039146 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" podStartSLOduration=2.039134689 podStartE2EDuration="2.039134689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:59.037956529 +0000 UTC m=+527.123412889" watchObservedRunningTime="2026-09-11 14:23:59.039134689 +0000 UTC m=+527.124591049" Sep 11 14:23:59.040530 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:23:59.040506 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-824qd\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="880692f5-a7a6-428c-b9be-bb543b5684ca" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-824qd" Sep 11 14:24:09.909426 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:09.909345 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:24:09.909426 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:09.909415 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" Sep 11 14:24:09.909911 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:09.909445 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5k6br" Sep 11 14:24:14.987294 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:14.987262 2578 generic.go:356] "Generic (PLEG): container finished" podID="ebc9d097-340d-4a73-ba10-1b716ff4b7c0" containerID="4aefc9485bab8c5128caf1ed8343b05726949cf609bddc910fc9bc47915ba9c4" exitCode=1 Sep 11 14:24:14.987673 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:14.987323 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" event={"ID":"ebc9d097-340d-4a73-ba10-1b716ff4b7c0","Type":"ContainerDied","Data":"4aefc9485bab8c5128caf1ed8343b05726949cf609bddc910fc9bc47915ba9c4"} Sep 11 14:24:14.987673 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:14.987609 2578 scope.go:117] "RemoveContainer" containerID="4aefc9485bab8c5128caf1ed8343b05726949cf609bddc910fc9bc47915ba9c4" Sep 11 14:24:15.994745 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:15.994701 2578 generic.go:356] "Generic (PLEG): container finished" podID="c245cb01-c3b2-4e88-9958-8987f62d91a8" containerID="05bec0f39043c1ce58c62082f46ed1f579f4864a69ef0e4daf86fe61f7c3a991" exitCode=1 Sep 11 14:24:15.995135 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:15.994775 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" event={"ID":"c245cb01-c3b2-4e88-9958-8987f62d91a8","Type":"ContainerDied","Data":"05bec0f39043c1ce58c62082f46ed1f579f4864a69ef0e4daf86fe61f7c3a991"} Sep 11 14:24:15.995211 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:15.995144 2578 scope.go:117] "RemoveContainer" containerID="05bec0f39043c1ce58c62082f46ed1f579f4864a69ef0e4daf86fe61f7c3a991" Sep 11 14:24:15.996826 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:15.996804 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" event={"ID":"ebc9d097-340d-4a73-ba10-1b716ff4b7c0","Type":"ContainerStarted","Data":"fd07625f54119868814c97943d849c14a7b1025bc06b93ca8c052a89532caf88"} Sep 11 14:24:15.997014 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:15.997000 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" Sep 11 14:24:17.004629 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:17.004598 2578 generic.go:356] "Generic (PLEG): container finished" podID="23e76c7f-aeae-4756-bb38-eba9de65e417" containerID="6cdbac991650f3f80858770648a8b1f2a3eccd2ce12c37049f04137523eb067c" exitCode=1 Sep 11 14:24:17.005049 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:17.004672 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" event={"ID":"23e76c7f-aeae-4756-bb38-eba9de65e417","Type":"ContainerDied","Data":"6cdbac991650f3f80858770648a8b1f2a3eccd2ce12c37049f04137523eb067c"} Sep 11 14:24:17.005049 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:17.005008 2578 scope.go:117] "RemoveContainer" containerID="6cdbac991650f3f80858770648a8b1f2a3eccd2ce12c37049f04137523eb067c" Sep 11 14:24:17.006638 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:17.006617 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" event={"ID":"c245cb01-c3b2-4e88-9958-8987f62d91a8","Type":"ContainerStarted","Data":"60c285909ca2ee76cbdf9cd774b1086f3e796000d62d3717396fa5c0360921d4"} Sep 11 14:24:17.006873 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:17.006854 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" Sep 11 14:24:17.008432 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:17.008413 2578 generic.go:356] "Generic (PLEG): container finished" podID="c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb" containerID="9c0e5d8bb4c8ccfe757841330f91db9150e4edfa282a57fade5f9ac2f7a3e10e" exitCode=1 Sep 11 14:24:17.008492 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:17.008478 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" event={"ID":"c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb","Type":"ContainerDied","Data":"9c0e5d8bb4c8ccfe757841330f91db9150e4edfa282a57fade5f9ac2f7a3e10e"} Sep 11 14:24:17.008718 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:17.008706 2578 scope.go:117] "RemoveContainer" containerID="9c0e5d8bb4c8ccfe757841330f91db9150e4edfa282a57fade5f9ac2f7a3e10e" Sep 11 14:24:18.016749 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:18.016699 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" event={"ID":"23e76c7f-aeae-4756-bb38-eba9de65e417","Type":"ContainerStarted","Data":"1032ec3fbdbb285a886984a7ca7933a35327034ad470d7e3d9b2c2ee80efed89"} Sep 11 14:24:18.017175 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:18.016919 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" Sep 11 14:24:18.018674 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:18.018650 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" event={"ID":"c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb","Type":"ContainerStarted","Data":"c4a67427931ecc6a130a665e9b3f44b59265861718b8047a98a89545b8ab5288"} Sep 11 14:24:18.018853 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:18.018837 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" Sep 11 14:24:19.024923 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:19.024890 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" Sep 11 14:24:29.026332 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:29.026298 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-7m8nt" Sep 11 14:24:30.031638 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:30.031608 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-9h7sk" Sep 11 14:24:38.129964 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:38.129929 2578 generic.go:356] "Generic (PLEG): container finished" podID="d616a45a-24e4-455f-a5b5-b487c7a4708d" containerID="9e657f504347fb52a456b1aabd78fe932ee8b0fd1794daffdd119fcb6ce3c503" exitCode=0 Sep 11 14:24:38.130370 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:38.129985 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" event={"ID":"d616a45a-24e4-455f-a5b5-b487c7a4708d","Type":"ContainerDied","Data":"9e657f504347fb52a456b1aabd78fe932ee8b0fd1794daffdd119fcb6ce3c503"} Sep 11 14:24:38.130370 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:38.130340 2578 scope.go:117] "RemoveContainer" containerID="9e657f504347fb52a456b1aabd78fe932ee8b0fd1794daffdd119fcb6ce3c503" Sep 11 14:24:39.137863 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:39.137829 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" event={"ID":"d616a45a-24e4-455f-a5b5-b487c7a4708d","Type":"ContainerStarted","Data":"630776e2b120de176b236b0c3f41892ec740f27a456ae08ef9df0a12c786239f"} Sep 11 14:24:39.138228 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:39.138071 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:24:47.011650 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:47.011613 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dk4tz" Sep 11 14:24:48.024398 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:48.024369 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pb8jz" Sep 11 14:24:50.145924 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:50.145884 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:24:51.425905 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:51.425869 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq"] Sep 11 14:24:51.426357 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:51.426065 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" podUID="b0e528e6-f8a9-4511-9a80-b68f6de1e83b" containerName="manager" containerID="cri-o://43eeb5f2daa7a24a1c2ea96cbbef013f893c3353e5735b5c59b591fb271f01b8" gracePeriod=10 Sep 11 14:24:51.685350 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:51.685283 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" Sep 11 14:24:51.825626 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:51.825587 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9txdl\" (UniqueName: \"kubernetes.io/projected/b0e528e6-f8a9-4511-9a80-b68f6de1e83b-kube-api-access-9txdl\") pod \"b0e528e6-f8a9-4511-9a80-b68f6de1e83b\" (UID: \"b0e528e6-f8a9-4511-9a80-b68f6de1e83b\") " Sep 11 14:24:51.825813 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:51.825641 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0e528e6-f8a9-4511-9a80-b68f6de1e83b-extensions-socket-volume\") pod \"b0e528e6-f8a9-4511-9a80-b68f6de1e83b\" (UID: \"b0e528e6-f8a9-4511-9a80-b68f6de1e83b\") " Sep 11 14:24:51.826035 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:51.826009 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e528e6-f8a9-4511-9a80-b68f6de1e83b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b0e528e6-f8a9-4511-9a80-b68f6de1e83b" (UID: "b0e528e6-f8a9-4511-9a80-b68f6de1e83b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:24:51.827787 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:51.827761 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e528e6-f8a9-4511-9a80-b68f6de1e83b-kube-api-access-9txdl" (OuterVolumeSpecName: "kube-api-access-9txdl") pod "b0e528e6-f8a9-4511-9a80-b68f6de1e83b" (UID: "b0e528e6-f8a9-4511-9a80-b68f6de1e83b"). InnerVolumeSpecName "kube-api-access-9txdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:24:51.926827 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:51.926791 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9txdl\" (UniqueName: \"kubernetes.io/projected/b0e528e6-f8a9-4511-9a80-b68f6de1e83b-kube-api-access-9txdl\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:24:51.926827 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:51.926821 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0e528e6-f8a9-4511-9a80-b68f6de1e83b-extensions-socket-volume\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:24:52.213192 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.213154 2578 generic.go:356] "Generic (PLEG): container finished" podID="b0e528e6-f8a9-4511-9a80-b68f6de1e83b" containerID="43eeb5f2daa7a24a1c2ea96cbbef013f893c3353e5735b5c59b591fb271f01b8" exitCode=0 Sep 11 14:24:52.213384 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.213202 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" Sep 11 14:24:52.213384 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.213239 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" event={"ID":"b0e528e6-f8a9-4511-9a80-b68f6de1e83b","Type":"ContainerDied","Data":"43eeb5f2daa7a24a1c2ea96cbbef013f893c3353e5735b5c59b591fb271f01b8"} Sep 11 14:24:52.213384 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.213296 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq" event={"ID":"b0e528e6-f8a9-4511-9a80-b68f6de1e83b","Type":"ContainerDied","Data":"213e18c1a884fe33f633114b1cc650ce26e34b84ee1e53c34f7d10aff4f3b13e"} Sep 11 14:24:52.213384 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.213321 2578 scope.go:117] "RemoveContainer" containerID="43eeb5f2daa7a24a1c2ea96cbbef013f893c3353e5735b5c59b591fb271f01b8" Sep 11 14:24:52.222649 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.222625 2578 scope.go:117] "RemoveContainer" containerID="43eeb5f2daa7a24a1c2ea96cbbef013f893c3353e5735b5c59b591fb271f01b8" Sep 11 14:24:52.222981 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:24:52.222960 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43eeb5f2daa7a24a1c2ea96cbbef013f893c3353e5735b5c59b591fb271f01b8\": container with ID starting with 43eeb5f2daa7a24a1c2ea96cbbef013f893c3353e5735b5c59b591fb271f01b8 not found: ID does not exist" containerID="43eeb5f2daa7a24a1c2ea96cbbef013f893c3353e5735b5c59b591fb271f01b8" Sep 11 14:24:52.223064 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.222990 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43eeb5f2daa7a24a1c2ea96cbbef013f893c3353e5735b5c59b591fb271f01b8"} err="failed to get container status \"43eeb5f2daa7a24a1c2ea96cbbef013f893c3353e5735b5c59b591fb271f01b8\": rpc error: code = NotFound desc = could not find container \"43eeb5f2daa7a24a1c2ea96cbbef013f893c3353e5735b5c59b591fb271f01b8\": container with ID starting with 43eeb5f2daa7a24a1c2ea96cbbef013f893c3353e5735b5c59b591fb271f01b8 not found: ID does not exist" Sep 11 14:24:52.227322 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.227300 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst"] Sep 11 14:24:52.227676 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.227664 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0e528e6-f8a9-4511-9a80-b68f6de1e83b" containerName="manager" Sep 11 14:24:52.227721 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.227677 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0e528e6-f8a9-4511-9a80-b68f6de1e83b" containerName="manager" Sep 11 14:24:52.227816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.227804 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0e528e6-f8a9-4511-9a80-b68f6de1e83b" containerName="manager" Sep 11 14:24:52.232088 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.232071 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" Sep 11 14:24:52.305643 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.305601 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst"] Sep 11 14:24:52.330876 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.330841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dcb07bc-03da-405b-b826-1d8c3ceb8d2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-66qst\" (UID: \"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" Sep 11 14:24:52.330876 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.330885 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7wln\" (UniqueName: \"kubernetes.io/projected/5dcb07bc-03da-405b-b826-1d8c3ceb8d2a-kube-api-access-g7wln\") pod \"kuadrant-operator-controller-manager-5769f99d9f-66qst\" (UID: \"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" Sep 11 14:24:52.405703 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.405668 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq"] Sep 11 14:24:52.431396 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.431358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dcb07bc-03da-405b-b826-1d8c3ceb8d2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-66qst\" (UID: \"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" Sep 11 14:24:52.431837 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.431414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7wln\" (UniqueName: \"kubernetes.io/projected/5dcb07bc-03da-405b-b826-1d8c3ceb8d2a-kube-api-access-g7wln\") pod \"kuadrant-operator-controller-manager-5769f99d9f-66qst\" (UID: \"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" Sep 11 14:24:52.431837 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.431768 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dcb07bc-03da-405b-b826-1d8c3ceb8d2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-66qst\" (UID: \"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" Sep 11 14:24:52.455329 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.455295 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-vqcvq"] Sep 11 14:24:52.464117 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.464043 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0e528e6-f8a9-4511-9a80-b68f6de1e83b" path="/var/lib/kubelet/pods/b0e528e6-f8a9-4511-9a80-b68f6de1e83b/volumes" Sep 11 14:24:52.478393 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.478357 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7wln\" (UniqueName: \"kubernetes.io/projected/5dcb07bc-03da-405b-b826-1d8c3ceb8d2a-kube-api-access-g7wln\") pod \"kuadrant-operator-controller-manager-5769f99d9f-66qst\" (UID: \"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" Sep 11 14:24:52.542308 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.542272 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" Sep 11 14:24:52.979816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:52.979782 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst"] Sep 11 14:24:52.986311 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:24:52.986281 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcb07bc_03da_405b_b826_1d8c3ceb8d2a.slice/crio-d2e2ce43188f1393c49d05ede5c9effe2a479f10f6f34f6129b5aef514f69715 WatchSource:0}: Error finding container d2e2ce43188f1393c49d05ede5c9effe2a479f10f6f34f6129b5aef514f69715: Status 404 returned error can't find the container with id d2e2ce43188f1393c49d05ede5c9effe2a479f10f6f34f6129b5aef514f69715 Sep 11 14:24:53.221550 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:53.221515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" event={"ID":"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a","Type":"ContainerStarted","Data":"f0de0621202c3b6008f3ae441b66cb0c6eca3a5612f63f5bd04eb3407e714372"} Sep 11 14:24:53.221712 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:53.221562 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" event={"ID":"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a","Type":"ContainerStarted","Data":"d2e2ce43188f1393c49d05ede5c9effe2a479f10f6f34f6129b5aef514f69715"} Sep 11 14:24:53.221712 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:53.221638 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" Sep 11 14:24:53.313413 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:24:53.313303 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" podStartSLOduration=1.3132860499999999 podStartE2EDuration="1.31328605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:24:53.311267688 +0000 UTC m=+581.396724054" watchObservedRunningTime="2026-09-11 14:24:53.31328605 +0000 UTC m=+581.398742409" Sep 11 14:25:04.229229 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:04.229199 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" Sep 11 14:25:04.325098 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:04.325052 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n"] Sep 11 14:25:04.325289 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:04.325265 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" podUID="d616a45a-24e4-455f-a5b5-b487c7a4708d" containerName="manager" containerID="cri-o://630776e2b120de176b236b0c3f41892ec740f27a456ae08ef9df0a12c786239f" gracePeriod=10 Sep 11 14:25:04.573878 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:04.573850 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:25:04.736499 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:04.736455 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d616a45a-24e4-455f-a5b5-b487c7a4708d-extensions-socket-volume\") pod \"d616a45a-24e4-455f-a5b5-b487c7a4708d\" (UID: \"d616a45a-24e4-455f-a5b5-b487c7a4708d\") " Sep 11 14:25:04.736691 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:04.736559 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2gqq\" (UniqueName: \"kubernetes.io/projected/d616a45a-24e4-455f-a5b5-b487c7a4708d-kube-api-access-h2gqq\") pod \"d616a45a-24e4-455f-a5b5-b487c7a4708d\" (UID: \"d616a45a-24e4-455f-a5b5-b487c7a4708d\") " Sep 11 14:25:04.736938 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:04.736914 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d616a45a-24e4-455f-a5b5-b487c7a4708d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d616a45a-24e4-455f-a5b5-b487c7a4708d" (UID: "d616a45a-24e4-455f-a5b5-b487c7a4708d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:25:04.738583 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:04.738561 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d616a45a-24e4-455f-a5b5-b487c7a4708d-kube-api-access-h2gqq" (OuterVolumeSpecName: "kube-api-access-h2gqq") pod "d616a45a-24e4-455f-a5b5-b487c7a4708d" (UID: "d616a45a-24e4-455f-a5b5-b487c7a4708d"). InnerVolumeSpecName "kube-api-access-h2gqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:25:04.837875 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:04.837837 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d616a45a-24e4-455f-a5b5-b487c7a4708d-extensions-socket-volume\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:25:04.837875 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:04.837870 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2gqq\" (UniqueName: \"kubernetes.io/projected/d616a45a-24e4-455f-a5b5-b487c7a4708d-kube-api-access-h2gqq\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:25:05.294629 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:05.294592 2578 generic.go:356] "Generic (PLEG): container finished" podID="d616a45a-24e4-455f-a5b5-b487c7a4708d" containerID="630776e2b120de176b236b0c3f41892ec740f27a456ae08ef9df0a12c786239f" exitCode=0 Sep 11 14:25:05.295130 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:05.294640 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" Sep 11 14:25:05.295130 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:05.294674 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" event={"ID":"d616a45a-24e4-455f-a5b5-b487c7a4708d","Type":"ContainerDied","Data":"630776e2b120de176b236b0c3f41892ec740f27a456ae08ef9df0a12c786239f"} Sep 11 14:25:05.295130 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:05.294746 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n" event={"ID":"d616a45a-24e4-455f-a5b5-b487c7a4708d","Type":"ContainerDied","Data":"0758eccfb4cc89543f2a61cb481b1807d69518cd8e01908d7c5e66b2259ea485"} Sep 11 14:25:05.295130 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:05.294773 2578 scope.go:117] "RemoveContainer" containerID="630776e2b120de176b236b0c3f41892ec740f27a456ae08ef9df0a12c786239f" Sep 11 14:25:05.304604 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:05.304585 2578 scope.go:117] "RemoveContainer" containerID="9e657f504347fb52a456b1aabd78fe932ee8b0fd1794daffdd119fcb6ce3c503" Sep 11 14:25:05.313122 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:05.313095 2578 scope.go:117] "RemoveContainer" containerID="630776e2b120de176b236b0c3f41892ec740f27a456ae08ef9df0a12c786239f" Sep 11 14:25:05.313385 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:25:05.313366 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"630776e2b120de176b236b0c3f41892ec740f27a456ae08ef9df0a12c786239f\": container with ID starting with 630776e2b120de176b236b0c3f41892ec740f27a456ae08ef9df0a12c786239f not found: ID does not exist" containerID="630776e2b120de176b236b0c3f41892ec740f27a456ae08ef9df0a12c786239f" Sep 11 14:25:05.313434 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:05.313395 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"630776e2b120de176b236b0c3f41892ec740f27a456ae08ef9df0a12c786239f"} err="failed to get container status \"630776e2b120de176b236b0c3f41892ec740f27a456ae08ef9df0a12c786239f\": rpc error: code = NotFound desc = could not find container \"630776e2b120de176b236b0c3f41892ec740f27a456ae08ef9df0a12c786239f\": container with ID starting with 630776e2b120de176b236b0c3f41892ec740f27a456ae08ef9df0a12c786239f not found: ID does not exist" Sep 11 14:25:05.313434 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:05.313413 2578 scope.go:117] "RemoveContainer" containerID="9e657f504347fb52a456b1aabd78fe932ee8b0fd1794daffdd119fcb6ce3c503" Sep 11 14:25:05.313658 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:25:05.313641 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e657f504347fb52a456b1aabd78fe932ee8b0fd1794daffdd119fcb6ce3c503\": container with ID starting with 9e657f504347fb52a456b1aabd78fe932ee8b0fd1794daffdd119fcb6ce3c503 not found: ID does not exist" containerID="9e657f504347fb52a456b1aabd78fe932ee8b0fd1794daffdd119fcb6ce3c503" Sep 11 14:25:05.313700 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:05.313664 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e657f504347fb52a456b1aabd78fe932ee8b0fd1794daffdd119fcb6ce3c503"} err="failed to get container status \"9e657f504347fb52a456b1aabd78fe932ee8b0fd1794daffdd119fcb6ce3c503\": rpc error: code = NotFound desc = could not find container \"9e657f504347fb52a456b1aabd78fe932ee8b0fd1794daffdd119fcb6ce3c503\": container with ID starting with 9e657f504347fb52a456b1aabd78fe932ee8b0fd1794daffdd119fcb6ce3c503 not found: ID does not exist" Sep 11 14:25:05.326121 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:05.326096 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n"] Sep 11 14:25:05.369120 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:05.369083 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-jhr6n"] Sep 11 14:25:06.467182 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:06.467147 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d616a45a-24e4-455f-a5b5-b487c7a4708d" path="/var/lib/kubelet/pods/d616a45a-24e4-455f-a5b5-b487c7a4708d/volumes" Sep 11 14:25:12.269816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:12.269790 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:25:12.272904 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:12.272876 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:25:23.940999 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:23.940967 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:25:23.941365 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:23.941331 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d616a45a-24e4-455f-a5b5-b487c7a4708d" containerName="manager" Sep 11 14:25:23.941365 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:23.941342 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d616a45a-24e4-455f-a5b5-b487c7a4708d" containerName="manager" Sep 11 14:25:23.941443 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:23.941410 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d616a45a-24e4-455f-a5b5-b487c7a4708d" containerName="manager" Sep 11 14:25:23.946173 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:23.946152 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-vw5gj" Sep 11 14:25:23.949075 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:23.949054 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 11 14:25:23.957456 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:23.957437 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:25:23.972681 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:23.972659 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:25:23.993509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:23.993485 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxrs\" (UniqueName: \"kubernetes.io/projected/ef9d0fa0-2947-4c16-a7f0-4b794b37c98a-kube-api-access-2vxrs\") pod \"limitador-limitador-84994f7ffc-vw5gj\" (UID: \"ef9d0fa0-2947-4c16-a7f0-4b794b37c98a\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vw5gj" Sep 11 14:25:23.993615 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:23.993558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ef9d0fa0-2947-4c16-a7f0-4b794b37c98a-config-file\") pod \"limitador-limitador-84994f7ffc-vw5gj\" (UID: \"ef9d0fa0-2947-4c16-a7f0-4b794b37c98a\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vw5gj" Sep 11 14:25:24.094173 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.094146 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxrs\" (UniqueName: \"kubernetes.io/projected/ef9d0fa0-2947-4c16-a7f0-4b794b37c98a-kube-api-access-2vxrs\") pod \"limitador-limitador-84994f7ffc-vw5gj\" (UID: \"ef9d0fa0-2947-4c16-a7f0-4b794b37c98a\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vw5gj" Sep 11 14:25:24.094319 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.094199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ef9d0fa0-2947-4c16-a7f0-4b794b37c98a-config-file\") pod \"limitador-limitador-84994f7ffc-vw5gj\" (UID: \"ef9d0fa0-2947-4c16-a7f0-4b794b37c98a\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vw5gj" Sep 11 14:25:24.094860 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.094842 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ef9d0fa0-2947-4c16-a7f0-4b794b37c98a-config-file\") pod \"limitador-limitador-84994f7ffc-vw5gj\" (UID: \"ef9d0fa0-2947-4c16-a7f0-4b794b37c98a\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vw5gj" Sep 11 14:25:24.104746 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.104695 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxrs\" (UniqueName: \"kubernetes.io/projected/ef9d0fa0-2947-4c16-a7f0-4b794b37c98a-kube-api-access-2vxrs\") pod \"limitador-limitador-84994f7ffc-vw5gj\" (UID: \"ef9d0fa0-2947-4c16-a7f0-4b794b37c98a\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vw5gj" Sep 11 14:25:24.257489 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.257420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-vw5gj" Sep 11 14:25:24.541773 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.541717 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:25:24.544561 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:25:24.544518 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9d0fa0_2947_4c16_a7f0_4b794b37c98a.slice/crio-315bcd15a6001a47095b58fc71092a531a89b3df8352c98bd36453dbb243f7d7 WatchSource:0}: Error finding container 315bcd15a6001a47095b58fc71092a531a89b3df8352c98bd36453dbb243f7d7: Status 404 returned error can't find the container with id 315bcd15a6001a47095b58fc71092a531a89b3df8352c98bd36453dbb243f7d7 Sep 11 14:25:24.858525 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.858496 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-2l2rw"] Sep 11 14:25:24.858883 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.858870 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d616a45a-24e4-455f-a5b5-b487c7a4708d" containerName="manager" Sep 11 14:25:24.858935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.858885 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d616a45a-24e4-455f-a5b5-b487c7a4708d" containerName="manager" Sep 11 14:25:24.858979 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.858956 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d616a45a-24e4-455f-a5b5-b487c7a4708d" containerName="manager" Sep 11 14:25:24.862994 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.862968 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2l2rw" Sep 11 14:25:24.866551 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.866531 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7mdtb\"" Sep 11 14:25:24.876527 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.876506 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2l2rw"] Sep 11 14:25:24.900375 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:24.900341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98v9k\" (UniqueName: \"kubernetes.io/projected/30cc6620-b117-486b-9262-e9da00fa5b9b-kube-api-access-98v9k\") pod \"authorino-59c974bd48-2l2rw\" (UID: \"30cc6620-b117-486b-9262-e9da00fa5b9b\") " pod="kuadrant-system/authorino-59c974bd48-2l2rw" Sep 11 14:25:25.001271 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:25.001243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98v9k\" (UniqueName: \"kubernetes.io/projected/30cc6620-b117-486b-9262-e9da00fa5b9b-kube-api-access-98v9k\") pod \"authorino-59c974bd48-2l2rw\" (UID: \"30cc6620-b117-486b-9262-e9da00fa5b9b\") " pod="kuadrant-system/authorino-59c974bd48-2l2rw" Sep 11 14:25:25.015745 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:25.015704 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98v9k\" (UniqueName: \"kubernetes.io/projected/30cc6620-b117-486b-9262-e9da00fa5b9b-kube-api-access-98v9k\") pod \"authorino-59c974bd48-2l2rw\" (UID: \"30cc6620-b117-486b-9262-e9da00fa5b9b\") " pod="kuadrant-system/authorino-59c974bd48-2l2rw" Sep 11 14:25:25.172955 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:25.172886 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2l2rw" Sep 11 14:25:25.413187 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:25.413131 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-vw5gj" event={"ID":"ef9d0fa0-2947-4c16-a7f0-4b794b37c98a","Type":"ContainerStarted","Data":"315bcd15a6001a47095b58fc71092a531a89b3df8352c98bd36453dbb243f7d7"} Sep 11 14:25:25.463017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:25.462979 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2l2rw"] Sep 11 14:25:25.463435 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:25:25.463406 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30cc6620_b117_486b_9262_e9da00fa5b9b.slice/crio-0943bec6cc0ce2abcd1b6feab60cc2c5b077cc1d2149f3c849303109ef9674e6 WatchSource:0}: Error finding container 0943bec6cc0ce2abcd1b6feab60cc2c5b077cc1d2149f3c849303109ef9674e6: Status 404 returned error can't find the container with id 0943bec6cc0ce2abcd1b6feab60cc2c5b077cc1d2149f3c849303109ef9674e6 Sep 11 14:25:26.424034 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:26.423997 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2l2rw" event={"ID":"30cc6620-b117-486b-9262-e9da00fa5b9b","Type":"ContainerStarted","Data":"0943bec6cc0ce2abcd1b6feab60cc2c5b077cc1d2149f3c849303109ef9674e6"} Sep 11 14:25:26.426038 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:26.426008 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-vw5gj" event={"ID":"ef9d0fa0-2947-4c16-a7f0-4b794b37c98a","Type":"ContainerStarted","Data":"5c3ba87b469dd7945f4bb1e6f7c9af6f9e667727835f6328083ecb40d8d46580"} Sep 11 14:25:26.426173 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:26.426081 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-vw5gj" Sep 11 14:25:26.459234 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:26.459185 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-vw5gj" podStartSLOduration=2.296798569 podStartE2EDuration="3.459166058s" totalImagesPullingTime="1.162367489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:25:26.456064935 +0000 UTC m=+614.541521298" watchObservedRunningTime="2026-09-11 14:25:26.459166058 +0000 UTC m=+614.544622421" Sep 11 14:25:27.433074 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:27.432990 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2l2rw" event={"ID":"30cc6620-b117-486b-9262-e9da00fa5b9b","Type":"ContainerStarted","Data":"86bc70eba25982f5901c962dc2cdd0619cdc8ec79aaa18042748b96952c8d61a"} Sep 11 14:25:27.469055 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:27.469006 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-2l2rw" podStartSLOduration=1.78862868 podStartE2EDuration="3.468988139s" totalImagesPullingTime="1.680359459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:25:27.466295061 +0000 UTC m=+615.551751422" watchObservedRunningTime="2026-09-11 14:25:27.468988139 +0000 UTC m=+615.554444533" Sep 11 14:25:37.434250 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:37.434183 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-vw5gj" Sep 11 14:25:56.229383 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.229343 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-cvn59"] Sep 11 14:25:56.232984 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.232966 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-cvn59" Sep 11 14:25:56.250703 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.250672 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-cvn59"] Sep 11 14:25:56.352615 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.352590 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf229\" (UniqueName: \"kubernetes.io/projected/19174720-b5ec-4fe5-a47b-8f7c428319ae-kube-api-access-zf229\") pod \"authorino-557f876f69-cvn59\" (UID: \"19174720-b5ec-4fe5-a47b-8f7c428319ae\") " pod="kuadrant-system/authorino-557f876f69-cvn59" Sep 11 14:25:56.453901 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.453868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf229\" (UniqueName: \"kubernetes.io/projected/19174720-b5ec-4fe5-a47b-8f7c428319ae-kube-api-access-zf229\") pod \"authorino-557f876f69-cvn59\" (UID: \"19174720-b5ec-4fe5-a47b-8f7c428319ae\") " pod="kuadrant-system/authorino-557f876f69-cvn59" Sep 11 14:25:56.464932 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.464909 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf229\" (UniqueName: \"kubernetes.io/projected/19174720-b5ec-4fe5-a47b-8f7c428319ae-kube-api-access-zf229\") pod \"authorino-557f876f69-cvn59\" (UID: \"19174720-b5ec-4fe5-a47b-8f7c428319ae\") " pod="kuadrant-system/authorino-557f876f69-cvn59" Sep 11 14:25:56.489914 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.489853 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-cvn59"] Sep 11 14:25:56.490063 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.490052 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-cvn59" Sep 11 14:25:56.546667 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.546583 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-869685c776-6sd96"] Sep 11 14:25:56.551832 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.551805 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-869685c776-6sd96" Sep 11 14:25:56.571662 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.571637 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-869685c776-6sd96"] Sep 11 14:25:56.607561 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.607535 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-869685c776-6sd96"] Sep 11 14:25:56.607775 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:25:56.607756 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dtp66], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-869685c776-6sd96" podUID="7e5e7a0d-b19a-4771-a6c4-d71a236648e4" Sep 11 14:25:56.642190 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.642158 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bddcb7ccd-fld6r"] Sep 11 14:25:56.645743 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.645716 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bddcb7ccd-fld6r" Sep 11 14:25:56.648966 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.648944 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 11 14:25:56.655570 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.655552 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtp66\" (UniqueName: \"kubernetes.io/projected/7e5e7a0d-b19a-4771-a6c4-d71a236648e4-kube-api-access-dtp66\") pod \"authorino-869685c776-6sd96\" (UID: \"7e5e7a0d-b19a-4771-a6c4-d71a236648e4\") " pod="kuadrant-system/authorino-869685c776-6sd96" Sep 11 14:25:56.698493 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.698468 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bddcb7ccd-fld6r"] Sep 11 14:25:56.756243 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.756180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtp66\" (UniqueName: \"kubernetes.io/projected/7e5e7a0d-b19a-4771-a6c4-d71a236648e4-kube-api-access-dtp66\") pod \"authorino-869685c776-6sd96\" (UID: \"7e5e7a0d-b19a-4771-a6c4-d71a236648e4\") " pod="kuadrant-system/authorino-869685c776-6sd96" Sep 11 14:25:56.756243 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.756229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxghn\" (UniqueName: \"kubernetes.io/projected/46491d8c-eb2d-4f91-8389-0aee7833a74a-kube-api-access-xxghn\") pod \"authorino-bddcb7ccd-fld6r\" (UID: \"46491d8c-eb2d-4f91-8389-0aee7833a74a\") " pod="kuadrant-system/authorino-bddcb7ccd-fld6r" Sep 11 14:25:56.756395 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.756273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46491d8c-eb2d-4f91-8389-0aee7833a74a-tls-cert\") pod \"authorino-bddcb7ccd-fld6r\" (UID: \"46491d8c-eb2d-4f91-8389-0aee7833a74a\") " pod="kuadrant-system/authorino-bddcb7ccd-fld6r" Sep 11 14:25:56.774073 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.774048 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-cvn59"] Sep 11 14:25:56.774460 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.774441 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtp66\" (UniqueName: \"kubernetes.io/projected/7e5e7a0d-b19a-4771-a6c4-d71a236648e4-kube-api-access-dtp66\") pod \"authorino-869685c776-6sd96\" (UID: \"7e5e7a0d-b19a-4771-a6c4-d71a236648e4\") " pod="kuadrant-system/authorino-869685c776-6sd96" Sep 11 14:25:56.775468 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:25:56.775444 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19174720_b5ec_4fe5_a47b_8f7c428319ae.slice/crio-63ad03808d082cf8daa635f27c6ebef6085da94efcf33a03913662238a4496af WatchSource:0}: Error finding container 63ad03808d082cf8daa635f27c6ebef6085da94efcf33a03913662238a4496af: Status 404 returned error can't find the container with id 63ad03808d082cf8daa635f27c6ebef6085da94efcf33a03913662238a4496af Sep 11 14:25:56.776825 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.776809 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:25:56.857295 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.857265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46491d8c-eb2d-4f91-8389-0aee7833a74a-tls-cert\") pod \"authorino-bddcb7ccd-fld6r\" (UID: \"46491d8c-eb2d-4f91-8389-0aee7833a74a\") " pod="kuadrant-system/authorino-bddcb7ccd-fld6r" Sep 11 14:25:56.857413 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.857346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxghn\" (UniqueName: \"kubernetes.io/projected/46491d8c-eb2d-4f91-8389-0aee7833a74a-kube-api-access-xxghn\") pod \"authorino-bddcb7ccd-fld6r\" (UID: \"46491d8c-eb2d-4f91-8389-0aee7833a74a\") " pod="kuadrant-system/authorino-bddcb7ccd-fld6r" Sep 11 14:25:56.859439 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.859420 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46491d8c-eb2d-4f91-8389-0aee7833a74a-tls-cert\") pod \"authorino-bddcb7ccd-fld6r\" (UID: \"46491d8c-eb2d-4f91-8389-0aee7833a74a\") " pod="kuadrant-system/authorino-bddcb7ccd-fld6r" Sep 11 14:25:56.876822 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.876803 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxghn\" (UniqueName: \"kubernetes.io/projected/46491d8c-eb2d-4f91-8389-0aee7833a74a-kube-api-access-xxghn\") pod \"authorino-bddcb7ccd-fld6r\" (UID: \"46491d8c-eb2d-4f91-8389-0aee7833a74a\") " pod="kuadrant-system/authorino-bddcb7ccd-fld6r" Sep 11 14:25:56.955411 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:56.955387 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bddcb7ccd-fld6r" Sep 11 14:25:57.238161 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:57.238135 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bddcb7ccd-fld6r"] Sep 11 14:25:57.238593 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:25:57.238570 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46491d8c_eb2d_4f91_8389_0aee7833a74a.slice/crio-f2883435d4ab9ac6b97eade6cdd23198be4af6fe6b27001d55c62d9f25ca6684 WatchSource:0}: Error finding container f2883435d4ab9ac6b97eade6cdd23198be4af6fe6b27001d55c62d9f25ca6684: Status 404 returned error can't find the container with id f2883435d4ab9ac6b97eade6cdd23198be4af6fe6b27001d55c62d9f25ca6684 Sep 11 14:25:57.597208 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:57.597177 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bddcb7ccd-fld6r" event={"ID":"46491d8c-eb2d-4f91-8389-0aee7833a74a","Type":"ContainerStarted","Data":"f2883435d4ab9ac6b97eade6cdd23198be4af6fe6b27001d55c62d9f25ca6684"} Sep 11 14:25:57.598423 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:57.598401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-869685c776-6sd96" Sep 11 14:25:57.598535 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:57.598402 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-cvn59" event={"ID":"19174720-b5ec-4fe5-a47b-8f7c428319ae","Type":"ContainerStarted","Data":"63ad03808d082cf8daa635f27c6ebef6085da94efcf33a03913662238a4496af"} Sep 11 14:25:57.606598 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:57.606581 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-869685c776-6sd96" Sep 11 14:25:57.666188 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:57.666169 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtp66\" (UniqueName: \"kubernetes.io/projected/7e5e7a0d-b19a-4771-a6c4-d71a236648e4-kube-api-access-dtp66\") pod \"7e5e7a0d-b19a-4771-a6c4-d71a236648e4\" (UID: \"7e5e7a0d-b19a-4771-a6c4-d71a236648e4\") " Sep 11 14:25:57.668179 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:57.668148 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5e7a0d-b19a-4771-a6c4-d71a236648e4-kube-api-access-dtp66" (OuterVolumeSpecName: "kube-api-access-dtp66") pod "7e5e7a0d-b19a-4771-a6c4-d71a236648e4" (UID: "7e5e7a0d-b19a-4771-a6c4-d71a236648e4"). InnerVolumeSpecName "kube-api-access-dtp66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:25:57.767233 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:57.767207 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtp66\" (UniqueName: \"kubernetes.io/projected/7e5e7a0d-b19a-4771-a6c4-d71a236648e4-kube-api-access-dtp66\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:25:58.605275 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:58.605235 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bddcb7ccd-fld6r" event={"ID":"46491d8c-eb2d-4f91-8389-0aee7833a74a","Type":"ContainerStarted","Data":"6f50f1c35e916e24b85cd3b142e03a6154f34aa87f70550ad3b7593368527f34"} Sep 11 14:25:58.606965 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:58.606939 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-cvn59" event={"ID":"19174720-b5ec-4fe5-a47b-8f7c428319ae","Type":"ContainerStarted","Data":"457059be25c2f9c644ba49cb4a665f204897ce7a07b272c4d6890193a509f24f"} Sep 11 14:25:58.607088 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:58.606969 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-cvn59" podUID="19174720-b5ec-4fe5-a47b-8f7c428319ae" containerName="authorino" containerID="cri-o://457059be25c2f9c644ba49cb4a665f204897ce7a07b272c4d6890193a509f24f" gracePeriod=30 Sep 11 14:25:58.607088 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:58.606981 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-869685c776-6sd96" Sep 11 14:25:58.625897 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:58.625852 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bddcb7ccd-fld6r" podStartSLOduration=1.852035713 podStartE2EDuration="2.625833254s" totalImagesPullingTime="773.797541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:25:58.625538993 +0000 UTC m=+646.710995366" watchObservedRunningTime="2026-09-11 14:25:58.625833254 +0000 UTC m=+646.711289614" Sep 11 14:25:58.659795 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:58.659767 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-869685c776-6sd96"] Sep 11 14:25:58.662212 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:58.662179 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2l2rw"] Sep 11 14:25:58.662377 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:58.662356 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-2l2rw" podUID="30cc6620-b117-486b-9262-e9da00fa5b9b" containerName="authorino" containerID="cri-o://86bc70eba25982f5901c962dc2cdd0619cdc8ec79aaa18042748b96952c8d61a" gracePeriod=30 Sep 11 14:25:58.669216 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:58.669196 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-869685c776-6sd96"] Sep 11 14:25:58.692134 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:58.692083 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-cvn59" podStartSLOduration=1.5243382460000001 podStartE2EDuration="2.692064838s" totalImagesPullingTime="1.167726592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:25:58.690780588 +0000 UTC m=+646.776236949" watchObservedRunningTime="2026-09-11 14:25:58.692064838 +0000 UTC m=+646.777521198" Sep 11 14:25:58.933583 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:58.933563 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-cvn59" Sep 11 14:25:58.936379 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:58.936362 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2l2rw" Sep 11 14:25:59.078252 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.078185 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf229\" (UniqueName: \"kubernetes.io/projected/19174720-b5ec-4fe5-a47b-8f7c428319ae-kube-api-access-zf229\") pod \"19174720-b5ec-4fe5-a47b-8f7c428319ae\" (UID: \"19174720-b5ec-4fe5-a47b-8f7c428319ae\") " Sep 11 14:25:59.078387 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.078261 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98v9k\" (UniqueName: \"kubernetes.io/projected/30cc6620-b117-486b-9262-e9da00fa5b9b-kube-api-access-98v9k\") pod \"30cc6620-b117-486b-9262-e9da00fa5b9b\" (UID: \"30cc6620-b117-486b-9262-e9da00fa5b9b\") " Sep 11 14:25:59.080184 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.080154 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cc6620-b117-486b-9262-e9da00fa5b9b-kube-api-access-98v9k" (OuterVolumeSpecName: "kube-api-access-98v9k") pod "30cc6620-b117-486b-9262-e9da00fa5b9b" (UID: "30cc6620-b117-486b-9262-e9da00fa5b9b"). InnerVolumeSpecName "kube-api-access-98v9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:25:59.080295 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.080228 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19174720-b5ec-4fe5-a47b-8f7c428319ae-kube-api-access-zf229" (OuterVolumeSpecName: "kube-api-access-zf229") pod "19174720-b5ec-4fe5-a47b-8f7c428319ae" (UID: "19174720-b5ec-4fe5-a47b-8f7c428319ae"). InnerVolumeSpecName "kube-api-access-zf229". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:25:59.178929 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.178905 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98v9k\" (UniqueName: \"kubernetes.io/projected/30cc6620-b117-486b-9262-e9da00fa5b9b-kube-api-access-98v9k\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:25:59.178929 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.178927 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf229\" (UniqueName: \"kubernetes.io/projected/19174720-b5ec-4fe5-a47b-8f7c428319ae-kube-api-access-zf229\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:25:59.614141 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.614109 2578 generic.go:356] "Generic (PLEG): container finished" podID="30cc6620-b117-486b-9262-e9da00fa5b9b" containerID="86bc70eba25982f5901c962dc2cdd0619cdc8ec79aaa18042748b96952c8d61a" exitCode=0 Sep 11 14:25:59.614554 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.614149 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2l2rw" Sep 11 14:25:59.614554 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.614184 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2l2rw" event={"ID":"30cc6620-b117-486b-9262-e9da00fa5b9b","Type":"ContainerDied","Data":"86bc70eba25982f5901c962dc2cdd0619cdc8ec79aaa18042748b96952c8d61a"} Sep 11 14:25:59.614554 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.614225 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2l2rw" event={"ID":"30cc6620-b117-486b-9262-e9da00fa5b9b","Type":"ContainerDied","Data":"0943bec6cc0ce2abcd1b6feab60cc2c5b077cc1d2149f3c849303109ef9674e6"} Sep 11 14:25:59.614554 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.614247 2578 scope.go:117] "RemoveContainer" containerID="86bc70eba25982f5901c962dc2cdd0619cdc8ec79aaa18042748b96952c8d61a" Sep 11 14:25:59.615626 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.615544 2578 generic.go:356] "Generic (PLEG): container finished" podID="19174720-b5ec-4fe5-a47b-8f7c428319ae" containerID="457059be25c2f9c644ba49cb4a665f204897ce7a07b272c4d6890193a509f24f" exitCode=0 Sep 11 14:25:59.615626 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.615602 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-cvn59" Sep 11 14:25:59.615762 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.615636 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-cvn59" event={"ID":"19174720-b5ec-4fe5-a47b-8f7c428319ae","Type":"ContainerDied","Data":"457059be25c2f9c644ba49cb4a665f204897ce7a07b272c4d6890193a509f24f"} Sep 11 14:25:59.615762 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.615666 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-cvn59" event={"ID":"19174720-b5ec-4fe5-a47b-8f7c428319ae","Type":"ContainerDied","Data":"63ad03808d082cf8daa635f27c6ebef6085da94efcf33a03913662238a4496af"} Sep 11 14:25:59.623374 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.623351 2578 scope.go:117] "RemoveContainer" containerID="86bc70eba25982f5901c962dc2cdd0619cdc8ec79aaa18042748b96952c8d61a" Sep 11 14:25:59.623712 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:25:59.623689 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bc70eba25982f5901c962dc2cdd0619cdc8ec79aaa18042748b96952c8d61a\": container with ID starting with 86bc70eba25982f5901c962dc2cdd0619cdc8ec79aaa18042748b96952c8d61a not found: ID does not exist" containerID="86bc70eba25982f5901c962dc2cdd0619cdc8ec79aaa18042748b96952c8d61a" Sep 11 14:25:59.623836 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.623722 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bc70eba25982f5901c962dc2cdd0619cdc8ec79aaa18042748b96952c8d61a"} err="failed to get container status \"86bc70eba25982f5901c962dc2cdd0619cdc8ec79aaa18042748b96952c8d61a\": rpc error: code = NotFound desc = could not find container \"86bc70eba25982f5901c962dc2cdd0619cdc8ec79aaa18042748b96952c8d61a\": container with ID starting with 86bc70eba25982f5901c962dc2cdd0619cdc8ec79aaa18042748b96952c8d61a not found: ID does not exist" Sep 11 14:25:59.623836 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.623768 2578 scope.go:117] "RemoveContainer" containerID="457059be25c2f9c644ba49cb4a665f204897ce7a07b272c4d6890193a509f24f" Sep 11 14:25:59.631845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.631829 2578 scope.go:117] "RemoveContainer" containerID="457059be25c2f9c644ba49cb4a665f204897ce7a07b272c4d6890193a509f24f" Sep 11 14:25:59.632079 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:25:59.632064 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457059be25c2f9c644ba49cb4a665f204897ce7a07b272c4d6890193a509f24f\": container with ID starting with 457059be25c2f9c644ba49cb4a665f204897ce7a07b272c4d6890193a509f24f not found: ID does not exist" containerID="457059be25c2f9c644ba49cb4a665f204897ce7a07b272c4d6890193a509f24f" Sep 11 14:25:59.632121 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.632085 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457059be25c2f9c644ba49cb4a665f204897ce7a07b272c4d6890193a509f24f"} err="failed to get container status \"457059be25c2f9c644ba49cb4a665f204897ce7a07b272c4d6890193a509f24f\": rpc error: code = NotFound desc = could not find container \"457059be25c2f9c644ba49cb4a665f204897ce7a07b272c4d6890193a509f24f\": container with ID starting with 457059be25c2f9c644ba49cb4a665f204897ce7a07b272c4d6890193a509f24f not found: ID does not exist" Sep 11 14:25:59.642544 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.642525 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-cvn59"] Sep 11 14:25:59.647918 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.647894 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-cvn59"] Sep 11 14:25:59.661197 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.661178 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2l2rw"] Sep 11 14:25:59.666176 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:25:59.666159 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2l2rw"] Sep 11 14:26:00.464166 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:00.464135 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19174720-b5ec-4fe5-a47b-8f7c428319ae" path="/var/lib/kubelet/pods/19174720-b5ec-4fe5-a47b-8f7c428319ae/volumes" Sep 11 14:26:00.464445 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:00.464432 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30cc6620-b117-486b-9262-e9da00fa5b9b" path="/var/lib/kubelet/pods/30cc6620-b117-486b-9262-e9da00fa5b9b/volumes" Sep 11 14:26:00.464710 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:00.464699 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e5e7a0d-b19a-4771-a6c4-d71a236648e4" path="/var/lib/kubelet/pods/7e5e7a0d-b19a-4771-a6c4-d71a236648e4/volumes" Sep 11 14:26:43.261014 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.260972 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jjtl7"] Sep 11 14:26:43.261631 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.261609 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19174720-b5ec-4fe5-a47b-8f7c428319ae" containerName="authorino" Sep 11 14:26:43.261677 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.261637 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="19174720-b5ec-4fe5-a47b-8f7c428319ae" containerName="authorino" Sep 11 14:26:43.261677 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.261674 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30cc6620-b117-486b-9262-e9da00fa5b9b" containerName="authorino" Sep 11 14:26:43.261757 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.261683 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30cc6620-b117-486b-9262-e9da00fa5b9b" containerName="authorino" Sep 11 14:26:43.261818 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.261803 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="19174720-b5ec-4fe5-a47b-8f7c428319ae" containerName="authorino" Sep 11 14:26:43.261856 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.261836 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30cc6620-b117-486b-9262-e9da00fa5b9b" containerName="authorino" Sep 11 14:26:43.264935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.264911 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-jjtl7" Sep 11 14:26:43.267943 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.267915 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 14:26:43.292578 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.292548 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jjtl7"] Sep 11 14:26:43.313362 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.313326 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb207b52-67df-47fb-a509-b099fafc0120-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jjtl7\" (UID: \"bb207b52-67df-47fb-a509-b099fafc0120\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jjtl7" Sep 11 14:26:43.313502 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.313393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftt4k\" (UniqueName: \"kubernetes.io/projected/bb207b52-67df-47fb-a509-b099fafc0120-kube-api-access-ftt4k\") pod \"payload-processing-5b54b7fbb7-jjtl7\" (UID: \"bb207b52-67df-47fb-a509-b099fafc0120\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jjtl7" Sep 11 14:26:43.322536 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.322499 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:26:43.414253 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.414210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb207b52-67df-47fb-a509-b099fafc0120-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jjtl7\" (UID: \"bb207b52-67df-47fb-a509-b099fafc0120\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jjtl7" Sep 11 14:26:43.414440 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.414279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftt4k\" (UniqueName: \"kubernetes.io/projected/bb207b52-67df-47fb-a509-b099fafc0120-kube-api-access-ftt4k\") pod \"payload-processing-5b54b7fbb7-jjtl7\" (UID: \"bb207b52-67df-47fb-a509-b099fafc0120\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jjtl7" Sep 11 14:26:43.415464 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.415428 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb207b52-67df-47fb-a509-b099fafc0120-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jjtl7\" (UID: \"bb207b52-67df-47fb-a509-b099fafc0120\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jjtl7" Sep 11 14:26:43.436613 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.436571 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftt4k\" (UniqueName: \"kubernetes.io/projected/bb207b52-67df-47fb-a509-b099fafc0120-kube-api-access-ftt4k\") pod \"payload-processing-5b54b7fbb7-jjtl7\" (UID: \"bb207b52-67df-47fb-a509-b099fafc0120\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jjtl7" Sep 11 14:26:43.577018 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.576936 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-jjtl7" Sep 11 14:26:43.877911 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:43.877882 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jjtl7"] Sep 11 14:26:43.878969 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:26:43.878942 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb207b52_67df_47fb_a509_b099fafc0120.slice/crio-a0219684377df60c8ef2573e3bf308a14cafc365c3d3c48c5aa9e0468e31e614 WatchSource:0}: Error finding container a0219684377df60c8ef2573e3bf308a14cafc365c3d3c48c5aa9e0468e31e614: Status 404 returned error can't find the container with id a0219684377df60c8ef2573e3bf308a14cafc365c3d3c48c5aa9e0468e31e614 Sep 11 14:26:44.883169 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:44.883109 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-jjtl7" event={"ID":"bb207b52-67df-47fb-a509-b099fafc0120","Type":"ContainerStarted","Data":"a0219684377df60c8ef2573e3bf308a14cafc365c3d3c48c5aa9e0468e31e614"} Sep 11 14:26:45.891593 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:45.891551 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-jjtl7" event={"ID":"bb207b52-67df-47fb-a509-b099fafc0120","Type":"ContainerStarted","Data":"ee281d2c4917913c9463bb4213bf255350773a9ebb1ca8ffeb5225b1c5b76031"} Sep 11 14:26:45.892038 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:45.891628 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-jjtl7" Sep 11 14:26:45.931390 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:45.931289 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-jjtl7" podStartSLOduration=1.154989895 podStartE2EDuration="2.931268788s" totalImagesPullingTime="1.776278893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:45.928673755 +0000 UTC m=+694.014130116" watchObservedRunningTime="2026-09-11 14:26:45.931268788 +0000 UTC m=+694.016725181" Sep 11 14:26:56.899640 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:26:56.899610 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-jjtl7" Sep 11 14:27:14.252152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:27:14.252060 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:27:29.832682 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:27:29.832650 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:27:44.633112 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:27:44.633081 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:27:58.421516 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:27:58.421469 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:28:01.648455 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.648424 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt"] Sep 11 14:28:01.652394 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.652365 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.657509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.657485 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 11 14:28:01.659341 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.658848 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 14:28:01.659341 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.658925 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m8wqn\"" Sep 11 14:28:01.659341 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.658967 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 14:28:01.676184 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.676163 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt"] Sep 11 14:28:01.760062 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.760027 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.760062 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.760062 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.760254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.760087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.760254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.760168 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.760254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.760202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ck9\" (UniqueName: \"kubernetes.io/projected/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-kube-api-access-h7ck9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.760254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.760235 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.861081 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.861050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.861238 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.861108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.861238 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.861127 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.861238 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.861167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.861238 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.861189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.861238 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.861205 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ck9\" (UniqueName: \"kubernetes.io/projected/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-kube-api-access-h7ck9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.861578 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.861549 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.861578 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.861570 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.861689 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.861599 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.863323 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.863300 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.863900 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.863884 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.872110 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.872086 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ck9\" (UniqueName: \"kubernetes.io/projected/e0a95587-8c5a-420e-a2bc-ac90c95a9ddc-kube-api-access-h7ck9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt\" (UID: \"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:01.963078 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:01.963004 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:02.248182 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:28:02.248146 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a95587_8c5a_420e_a2bc_ac90c95a9ddc.slice/crio-0ae600563b7e7188e45ff6ee37c24a9a3255c1b3f5e467628d87d54aab39e1a2 WatchSource:0}: Error finding container 0ae600563b7e7188e45ff6ee37c24a9a3255c1b3f5e467628d87d54aab39e1a2: Status 404 returned error can't find the container with id 0ae600563b7e7188e45ff6ee37c24a9a3255c1b3f5e467628d87d54aab39e1a2 Sep 11 14:28:02.249496 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:02.249466 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt"] Sep 11 14:28:02.326691 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:02.326660 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" event={"ID":"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc","Type":"ContainerStarted","Data":"0ae600563b7e7188e45ff6ee37c24a9a3255c1b3f5e467628d87d54aab39e1a2"} Sep 11 14:28:02.853264 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:02.853231 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:28:04.341820 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:04.341782 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" event={"ID":"e0a95587-8c5a-420e-a2bc-ac90c95a9ddc","Type":"ContainerStarted","Data":"bee2a874ce0e361541efbcaf827f76306f4b90e6c2e8f86af89559ff19df2e5c"} Sep 11 14:28:04.378403 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:04.378348 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" podStartSLOduration=2.349992683 podStartE2EDuration="3.378332265s" totalImagesPullingTime="1.028339582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:04.376222621 +0000 UTC m=+772.461678982" watchObservedRunningTime="2026-09-11 14:28:04.378332265 +0000 UTC m=+772.463788683" Sep 11 14:28:11.964036 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:11.963997 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:11.964444 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:11.964203 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:11.976158 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:11.976134 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:12.399285 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:12.399256 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt" Sep 11 14:28:13.929505 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:13.929472 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:28:29.350019 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:28:29.349979 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:30:01.765638 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:01.765597 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jxl6h"] Sep 11 14:30:01.769262 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:01.769239 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jxl6h" Sep 11 14:30:01.773838 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:01.773818 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\"" Sep 11 14:30:01.780533 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:01.780512 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jxl6h"] Sep 11 14:30:01.807032 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:01.807014 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jxl6h" Sep 11 14:30:02.085543 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:02.085519 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jxl6h"] Sep 11 14:30:02.087318 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:30:02.087272 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989739d0_7c12_4d1a_b6c5_731eb4d673e4.slice/crio-ce3e734828226919f8472e6c5c462a77045be20a652ee29d20100f6bd54b9f4e WatchSource:0}: Error finding container ce3e734828226919f8472e6c5c462a77045be20a652ee29d20100f6bd54b9f4e: Status 404 returned error can't find the container with id ce3e734828226919f8472e6c5c462a77045be20a652ee29d20100f6bd54b9f4e Sep 11 14:30:03.013331 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:03.013290 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jxl6h" event={"ID":"989739d0-7c12-4d1a-b6c5-731eb4d673e4","Type":"ContainerStarted","Data":"ce3e734828226919f8472e6c5c462a77045be20a652ee29d20100f6bd54b9f4e"} Sep 11 14:30:07.037174 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:07.037097 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jxl6h" event={"ID":"989739d0-7c12-4d1a-b6c5-731eb4d673e4","Type":"ContainerStarted","Data":"3fbf50c25c6345494ddcf41ec8f7ee53cece7e71ea6f3a04fa37341198945dd1"} Sep 11 14:30:07.064897 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:07.064849 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jxl6h" podStartSLOduration=1.387403376 podStartE2EDuration="6.064833962s" totalImagesPullingTime="4.677430586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:07.063389559 +0000 UTC m=+895.148845918" watchObservedRunningTime="2026-09-11 14:30:07.064833962 +0000 UTC m=+895.150290346" Sep 11 14:30:08.044715 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:08.044676 2578 generic.go:356] "Generic (PLEG): container finished" podID="989739d0-7c12-4d1a-b6c5-731eb4d673e4" containerID="3fbf50c25c6345494ddcf41ec8f7ee53cece7e71ea6f3a04fa37341198945dd1" exitCode=0 Sep 11 14:30:08.045129 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:08.044767 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jxl6h" event={"ID":"989739d0-7c12-4d1a-b6c5-731eb4d673e4","Type":"ContainerDied","Data":"3fbf50c25c6345494ddcf41ec8f7ee53cece7e71ea6f3a04fa37341198945dd1"} Sep 11 14:30:09.175897 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:09.175873 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jxl6h" Sep 11 14:30:10.058084 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:10.058050 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jxl6h" Sep 11 14:30:10.058283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:10.058103 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818950-jxl6h" event={"ID":"989739d0-7c12-4d1a-b6c5-731eb4d673e4","Type":"ContainerDied","Data":"ce3e734828226919f8472e6c5c462a77045be20a652ee29d20100f6bd54b9f4e"} Sep 11 14:30:10.058283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:10.058155 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce3e734828226919f8472e6c5c462a77045be20a652ee29d20100f6bd54b9f4e" Sep 11 14:30:12.320816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:12.320789 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:30:12.321537 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:12.321514 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:30:31.849179 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:31.849141 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9cff8d495-dnpkm"] Sep 11 14:30:31.849885 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:31.849838 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="989739d0-7c12-4d1a-b6c5-731eb4d673e4" containerName="cleanup" Sep 11 14:30:31.849885 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:31.849861 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="989739d0-7c12-4d1a-b6c5-731eb4d673e4" containerName="cleanup" Sep 11 14:30:31.850013 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:31.849978 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="989739d0-7c12-4d1a-b6c5-731eb4d673e4" containerName="cleanup" Sep 11 14:30:31.856324 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:31.856297 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9cff8d495-dnpkm" Sep 11 14:30:31.868260 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:31.868236 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9cff8d495-dnpkm"] Sep 11 14:30:31.986421 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:31.986390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xcdt\" (UniqueName: \"kubernetes.io/projected/e2aa03ae-36a5-49a7-9395-02a822a9a3d1-kube-api-access-6xcdt\") pod \"authorino-9cff8d495-dnpkm\" (UID: \"e2aa03ae-36a5-49a7-9395-02a822a9a3d1\") " pod="kuadrant-system/authorino-9cff8d495-dnpkm" Sep 11 14:30:31.986608 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:31.986458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2aa03ae-36a5-49a7-9395-02a822a9a3d1-tls-cert\") pod \"authorino-9cff8d495-dnpkm\" (UID: \"e2aa03ae-36a5-49a7-9395-02a822a9a3d1\") " pod="kuadrant-system/authorino-9cff8d495-dnpkm" Sep 11 14:30:32.087385 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:32.087354 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2aa03ae-36a5-49a7-9395-02a822a9a3d1-tls-cert\") pod \"authorino-9cff8d495-dnpkm\" (UID: \"e2aa03ae-36a5-49a7-9395-02a822a9a3d1\") " pod="kuadrant-system/authorino-9cff8d495-dnpkm" Sep 11 14:30:32.087538 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:32.087413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xcdt\" (UniqueName: \"kubernetes.io/projected/e2aa03ae-36a5-49a7-9395-02a822a9a3d1-kube-api-access-6xcdt\") pod \"authorino-9cff8d495-dnpkm\" (UID: \"e2aa03ae-36a5-49a7-9395-02a822a9a3d1\") " pod="kuadrant-system/authorino-9cff8d495-dnpkm" Sep 11 14:30:32.089923 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:32.089899 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2aa03ae-36a5-49a7-9395-02a822a9a3d1-tls-cert\") pod \"authorino-9cff8d495-dnpkm\" (UID: \"e2aa03ae-36a5-49a7-9395-02a822a9a3d1\") " pod="kuadrant-system/authorino-9cff8d495-dnpkm" Sep 11 14:30:32.099808 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:32.099754 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xcdt\" (UniqueName: \"kubernetes.io/projected/e2aa03ae-36a5-49a7-9395-02a822a9a3d1-kube-api-access-6xcdt\") pod \"authorino-9cff8d495-dnpkm\" (UID: \"e2aa03ae-36a5-49a7-9395-02a822a9a3d1\") " pod="kuadrant-system/authorino-9cff8d495-dnpkm" Sep 11 14:30:32.166266 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:32.166236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9cff8d495-dnpkm" Sep 11 14:30:32.457074 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:32.457042 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9cff8d495-dnpkm"] Sep 11 14:30:32.458851 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:30:32.458819 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2aa03ae_36a5_49a7_9395_02a822a9a3d1.slice/crio-3e047ef3bfcbd132851a963e036b69b8c0febbbd386d5c9fc9c1e5a6be93f46e WatchSource:0}: Error finding container 3e047ef3bfcbd132851a963e036b69b8c0febbbd386d5c9fc9c1e5a6be93f46e: Status 404 returned error can't find the container with id 3e047ef3bfcbd132851a963e036b69b8c0febbbd386d5c9fc9c1e5a6be93f46e Sep 11 14:30:33.192980 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:33.192944 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9cff8d495-dnpkm" event={"ID":"e2aa03ae-36a5-49a7-9395-02a822a9a3d1","Type":"ContainerStarted","Data":"a647c6b6c254073e99a8d1630b2978b38a8dd516d42d889ea52011ea1f3acb7e"} Sep 11 14:30:33.193363 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:33.192996 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9cff8d495-dnpkm" event={"ID":"e2aa03ae-36a5-49a7-9395-02a822a9a3d1","Type":"ContainerStarted","Data":"3e047ef3bfcbd132851a963e036b69b8c0febbbd386d5c9fc9c1e5a6be93f46e"} Sep 11 14:30:33.229075 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:33.229026 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9cff8d495-dnpkm" podStartSLOduration=1.671390629 podStartE2EDuration="2.229011078s" totalImagesPullingTime="557.620449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:33.225248938 +0000 UTC m=+921.310705298" watchObservedRunningTime="2026-09-11 14:30:33.229011078 +0000 UTC m=+921.314467437" Sep 11 14:30:33.297876 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:33.297799 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bddcb7ccd-fld6r"] Sep 11 14:30:33.298037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:33.297971 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-bddcb7ccd-fld6r" podUID="46491d8c-eb2d-4f91-8389-0aee7833a74a" containerName="authorino" containerID="cri-o://6f50f1c35e916e24b85cd3b142e03a6154f34aa87f70550ad3b7593368527f34" gracePeriod=30 Sep 11 14:30:33.561203 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:33.561144 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bddcb7ccd-fld6r" Sep 11 14:30:33.700854 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:33.700816 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46491d8c-eb2d-4f91-8389-0aee7833a74a-tls-cert\") pod \"46491d8c-eb2d-4f91-8389-0aee7833a74a\" (UID: \"46491d8c-eb2d-4f91-8389-0aee7833a74a\") " Sep 11 14:30:33.701013 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:33.700867 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxghn\" (UniqueName: \"kubernetes.io/projected/46491d8c-eb2d-4f91-8389-0aee7833a74a-kube-api-access-xxghn\") pod \"46491d8c-eb2d-4f91-8389-0aee7833a74a\" (UID: \"46491d8c-eb2d-4f91-8389-0aee7833a74a\") " Sep 11 14:30:33.702979 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:33.702944 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46491d8c-eb2d-4f91-8389-0aee7833a74a-kube-api-access-xxghn" (OuterVolumeSpecName: "kube-api-access-xxghn") pod "46491d8c-eb2d-4f91-8389-0aee7833a74a" (UID: "46491d8c-eb2d-4f91-8389-0aee7833a74a"). InnerVolumeSpecName "kube-api-access-xxghn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:30:33.710945 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:33.710917 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46491d8c-eb2d-4f91-8389-0aee7833a74a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "46491d8c-eb2d-4f91-8389-0aee7833a74a" (UID: "46491d8c-eb2d-4f91-8389-0aee7833a74a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:30:33.801935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:33.801896 2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46491d8c-eb2d-4f91-8389-0aee7833a74a-tls-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:30:33.801935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:33.801928 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxghn\" (UniqueName: \"kubernetes.io/projected/46491d8c-eb2d-4f91-8389-0aee7833a74a-kube-api-access-xxghn\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:30:34.203748 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:34.203687 2578 generic.go:356] "Generic (PLEG): container finished" podID="46491d8c-eb2d-4f91-8389-0aee7833a74a" containerID="6f50f1c35e916e24b85cd3b142e03a6154f34aa87f70550ad3b7593368527f34" exitCode=0 Sep 11 14:30:34.204429 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:34.204402 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bddcb7ccd-fld6r" Sep 11 14:30:34.213278 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:34.213237 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bddcb7ccd-fld6r" event={"ID":"46491d8c-eb2d-4f91-8389-0aee7833a74a","Type":"ContainerDied","Data":"6f50f1c35e916e24b85cd3b142e03a6154f34aa87f70550ad3b7593368527f34"} Sep 11 14:30:34.213440 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:34.213320 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bddcb7ccd-fld6r" event={"ID":"46491d8c-eb2d-4f91-8389-0aee7833a74a","Type":"ContainerDied","Data":"f2883435d4ab9ac6b97eade6cdd23198be4af6fe6b27001d55c62d9f25ca6684"} Sep 11 14:30:34.213440 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:34.213345 2578 scope.go:117] "RemoveContainer" containerID="6f50f1c35e916e24b85cd3b142e03a6154f34aa87f70550ad3b7593368527f34" Sep 11 14:30:34.224478 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:34.224463 2578 scope.go:117] "RemoveContainer" containerID="6f50f1c35e916e24b85cd3b142e03a6154f34aa87f70550ad3b7593368527f34" Sep 11 14:30:34.224765 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:30:34.224745 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f50f1c35e916e24b85cd3b142e03a6154f34aa87f70550ad3b7593368527f34\": container with ID starting with 6f50f1c35e916e24b85cd3b142e03a6154f34aa87f70550ad3b7593368527f34 not found: ID does not exist" containerID="6f50f1c35e916e24b85cd3b142e03a6154f34aa87f70550ad3b7593368527f34" Sep 11 14:30:34.224831 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:34.224772 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f50f1c35e916e24b85cd3b142e03a6154f34aa87f70550ad3b7593368527f34"} err="failed to get container status \"6f50f1c35e916e24b85cd3b142e03a6154f34aa87f70550ad3b7593368527f34\": rpc error: code = NotFound desc = could not find container \"6f50f1c35e916e24b85cd3b142e03a6154f34aa87f70550ad3b7593368527f34\": container with ID starting with 6f50f1c35e916e24b85cd3b142e03a6154f34aa87f70550ad3b7593368527f34 not found: ID does not exist" Sep 11 14:30:34.252557 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:34.252528 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bddcb7ccd-fld6r"] Sep 11 14:30:34.264813 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:34.264785 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-bddcb7ccd-fld6r"] Sep 11 14:30:34.475290 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:34.475247 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46491d8c-eb2d-4f91-8389-0aee7833a74a" path="/var/lib/kubelet/pods/46491d8c-eb2d-4f91-8389-0aee7833a74a/volumes" Sep 11 14:30:58.209252 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:58.209214 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8025-952939"] Sep 11 14:30:58.211530 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:58.209616 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46491d8c-eb2d-4f91-8389-0aee7833a74a" containerName="authorino" Sep 11 14:30:58.211530 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:58.209628 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46491d8c-eb2d-4f91-8389-0aee7833a74a" containerName="authorino" Sep 11 14:30:58.211530 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:58.209693 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="46491d8c-eb2d-4f91-8389-0aee7833a74a" containerName="authorino" Sep 11 14:30:58.212658 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:58.212643 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8025-952939" Sep 11 14:30:58.217824 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:58.217805 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7pn49\"" Sep 11 14:30:58.238416 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:58.238385 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8025-952939"] Sep 11 14:30:58.312530 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:58.312494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxww\" (UniqueName: \"kubernetes.io/projected/187b6e47-eb61-45a1-b254-58081f044712-kube-api-access-tqxww\") pod \"test-curl-8025-952939\" (UID: \"187b6e47-eb61-45a1-b254-58081f044712\") " pod="opendatahub/test-curl-8025-952939" Sep 11 14:30:58.413909 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:58.413867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxww\" (UniqueName: \"kubernetes.io/projected/187b6e47-eb61-45a1-b254-58081f044712-kube-api-access-tqxww\") pod \"test-curl-8025-952939\" (UID: \"187b6e47-eb61-45a1-b254-58081f044712\") " pod="opendatahub/test-curl-8025-952939" Sep 11 14:30:58.424900 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:58.424867 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxww\" (UniqueName: \"kubernetes.io/projected/187b6e47-eb61-45a1-b254-58081f044712-kube-api-access-tqxww\") pod \"test-curl-8025-952939\" (UID: \"187b6e47-eb61-45a1-b254-58081f044712\") " pod="opendatahub/test-curl-8025-952939" Sep 11 14:30:58.522568 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:58.522483 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8025-952939" Sep 11 14:30:58.807162 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:58.807088 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8025-952939"] Sep 11 14:30:58.807822 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:58.807788 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:30:59.353006 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:30:59.352963 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8025-952939" event={"ID":"187b6e47-eb61-45a1-b254-58081f044712","Type":"ContainerStarted","Data":"4a81ca67317250e380bd3dc7d078ce8408994f87c0c764e68abf29f9d93f19c1"} Sep 11 14:31:00.033557 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:00.033526 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:31:10.678147 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.678096 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn"] Sep 11 14:31:10.687338 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.687310 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:10.691960 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.691555 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7cfns\"" Sep 11 14:31:10.691960 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.691616 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 14:31:10.691960 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.691553 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:31:10.691960 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.691815 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1b6b934c-serving-cert\"" Sep 11 14:31:10.692636 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.692480 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 14:31:10.696504 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.696477 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn"] Sep 11 14:31:10.723927 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.723898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c26493f-6480-4559-a099-6090ac4830a0-openshift-service-ca\") pod \"maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:10.724075 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.723936 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-metrics-certs\") pod \"maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:10.724075 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.723985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9bzp\" (UniqueName: \"kubernetes.io/projected/8c26493f-6480-4559-a099-6090ac4830a0-kube-api-access-w9bzp\") pod \"maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:10.724075 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.724014 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-maas-api-tls\") pod \"maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:10.825139 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.825105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c26493f-6480-4559-a099-6090ac4830a0-openshift-service-ca\") pod \"maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:10.825139 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.825151 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-metrics-certs\") pod \"maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:10.825376 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.825203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9bzp\" (UniqueName: \"kubernetes.io/projected/8c26493f-6480-4559-a099-6090ac4830a0-kube-api-access-w9bzp\") pod \"maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:10.825376 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.825233 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-maas-api-tls\") pod \"maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:10.825487 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:10.825399 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-serving-cert: secret "maas-api-e2e-ait-1b6b934c-serving-cert" not found Sep 11 14:31:10.825487 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:10.825483 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-maas-api-tls podName:8c26493f-6480-4559-a099-6090ac4830a0 nodeName:}" failed. No retries permitted until 2026-09-11 14:31:11.325463318 +0000 UTC m=+959.410919670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-maas-api-tls") pod "maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" (UID: "8c26493f-6480-4559-a099-6090ac4830a0") : secret "maas-api-e2e-ait-1b6b934c-serving-cert" not found Sep 11 14:31:10.825928 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.825896 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c26493f-6480-4559-a099-6090ac4830a0-openshift-service-ca\") pod \"maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:10.827834 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.827807 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-metrics-certs\") pod \"maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:10.844425 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:10.844395 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9bzp\" (UniqueName: \"kubernetes.io/projected/8c26493f-6480-4559-a099-6090ac4830a0-kube-api-access-w9bzp\") pod \"maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:11.330094 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.330062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-maas-api-tls\") pod \"maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:11.332188 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.332168 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-maas-api-tls\") pod \"maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:11.434171 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.434123 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8025-952939" event={"ID":"187b6e47-eb61-45a1-b254-58081f044712","Type":"ContainerStarted","Data":"abc4686f2559c909444172f55946c5578ec4b5273b3e96f5600e69ec34b3abfd"} Sep 11 14:31:11.471570 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.471522 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8025-952939" podStartSLOduration=1.032684051 podStartE2EDuration="13.471507507s" totalImagesPullingTime="12.438823456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:11.465388312 +0000 UTC m=+959.550844671" watchObservedRunningTime="2026-09-11 14:31:11.471507507 +0000 UTC m=+959.556963867" Sep 11 14:31:11.604615 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.604583 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:11.710226 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.710190 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx"] Sep 11 14:31:11.714864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.714842 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" Sep 11 14:31:11.718063 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.718040 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1b6b934c\"" Sep 11 14:31:11.737928 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.737899 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx"] Sep 11 14:31:11.835983 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.835927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6383be-e171-4a9f-b33a-c79d01d42026-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx\" (UID: \"ea6383be-e171-4a9f-b33a-c79d01d42026\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" Sep 11 14:31:11.836149 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.836034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jlhz\" (UniqueName: \"kubernetes.io/projected/ea6383be-e171-4a9f-b33a-c79d01d42026-kube-api-access-2jlhz\") pod \"payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx\" (UID: \"ea6383be-e171-4a9f-b33a-c79d01d42026\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" Sep 11 14:31:11.894819 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.894793 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn"] Sep 11 14:31:11.896138 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:31:11.896108 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c26493f_6480_4559_a099_6090ac4830a0.slice/crio-8220fe48221f06eb480217ef7a7317d11d55b34e8b2df7f9ce6fe2becfbe2719 WatchSource:0}: Error finding container 8220fe48221f06eb480217ef7a7317d11d55b34e8b2df7f9ce6fe2becfbe2719: Status 404 returned error can't find the container with id 8220fe48221f06eb480217ef7a7317d11d55b34e8b2df7f9ce6fe2becfbe2719 Sep 11 14:31:11.937227 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.937200 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jlhz\" (UniqueName: \"kubernetes.io/projected/ea6383be-e171-4a9f-b33a-c79d01d42026-kube-api-access-2jlhz\") pod \"payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx\" (UID: \"ea6383be-e171-4a9f-b33a-c79d01d42026\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" Sep 11 14:31:11.937342 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.937264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6383be-e171-4a9f-b33a-c79d01d42026-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx\" (UID: \"ea6383be-e171-4a9f-b33a-c79d01d42026\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" Sep 11 14:31:11.937930 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.937911 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6383be-e171-4a9f-b33a-c79d01d42026-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx\" (UID: \"ea6383be-e171-4a9f-b33a-c79d01d42026\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" Sep 11 14:31:11.948545 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:11.948521 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jlhz\" (UniqueName: \"kubernetes.io/projected/ea6383be-e171-4a9f-b33a-c79d01d42026-kube-api-access-2jlhz\") pod \"payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx\" (UID: \"ea6383be-e171-4a9f-b33a-c79d01d42026\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" Sep 11 14:31:12.026062 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.026038 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" Sep 11 14:31:12.221975 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.221892 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8025-952939"] Sep 11 14:31:12.234604 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.234577 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8025-952939"] Sep 11 14:31:12.309091 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.309065 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx"] Sep 11 14:31:12.310883 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:31:12.310855 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6383be_e171_4a9f_b33a_c79d01d42026.slice/crio-debc9634a049c2b22eedebffb2351f341b672fde9c5c79a2281263b82ba42dd6 WatchSource:0}: Error finding container debc9634a049c2b22eedebffb2351f341b672fde9c5c79a2281263b82ba42dd6: Status 404 returned error can't find the container with id debc9634a049c2b22eedebffb2351f341b672fde9c5c79a2281263b82ba42dd6 Sep 11 14:31:12.386376 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.386350 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8025-f5f4e6"] Sep 11 14:31:12.386822 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.386806 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="187b6e47-eb61-45a1-b254-58081f044712" containerName="test-curl-8025-952939" Sep 11 14:31:12.386943 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.386825 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="187b6e47-eb61-45a1-b254-58081f044712" containerName="test-curl-8025-952939" Sep 11 14:31:12.386943 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.386914 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="187b6e47-eb61-45a1-b254-58081f044712" containerName="test-curl-8025-952939" Sep 11 14:31:12.389929 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.389913 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8025-f5f4e6" Sep 11 14:31:12.418071 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.418042 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8025-f5f4e6"] Sep 11 14:31:12.440120 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.440090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cpms\" (UniqueName: \"kubernetes.io/projected/d77b337a-74a3-4c1f-8d5a-349e2b43e9f3-kube-api-access-9cpms\") pod \"test-curl-8025-f5f4e6\" (UID: \"d77b337a-74a3-4c1f-8d5a-349e2b43e9f3\") " pod="opendatahub/test-curl-8025-f5f4e6" Sep 11 14:31:12.441940 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.441889 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" event={"ID":"ea6383be-e171-4a9f-b33a-c79d01d42026","Type":"ContainerStarted","Data":"65fa3132469dbbc51e4f1e3eb19b77c46da131bd9a0f3c4dd58648e8ed3ce48f"} Sep 11 14:31:12.442074 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.441954 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" Sep 11 14:31:12.442074 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.441977 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" event={"ID":"ea6383be-e171-4a9f-b33a-c79d01d42026","Type":"ContainerStarted","Data":"debc9634a049c2b22eedebffb2351f341b672fde9c5c79a2281263b82ba42dd6"} Sep 11 14:31:12.443270 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.443247 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" event={"ID":"8c26493f-6480-4559-a099-6090ac4830a0","Type":"ContainerStarted","Data":"8220fe48221f06eb480217ef7a7317d11d55b34e8b2df7f9ce6fe2becfbe2719"} Sep 11 14:31:12.443387 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:12.443352 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8025-952939" container="test-curl-8025-952939" Sep 11 14:31:12.471162 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.471109 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" podStartSLOduration=1.471089482 podStartE2EDuration="1.471089482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:12.470813225 +0000 UTC m=+960.556269585" watchObservedRunningTime="2026-09-11 14:31:12.471089482 +0000 UTC m=+960.556545839" Sep 11 14:31:12.540757 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.540657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cpms\" (UniqueName: \"kubernetes.io/projected/d77b337a-74a3-4c1f-8d5a-349e2b43e9f3-kube-api-access-9cpms\") pod \"test-curl-8025-f5f4e6\" (UID: \"d77b337a-74a3-4c1f-8d5a-349e2b43e9f3\") " pod="opendatahub/test-curl-8025-f5f4e6" Sep 11 14:31:12.552483 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.552431 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cpms\" (UniqueName: \"kubernetes.io/projected/d77b337a-74a3-4c1f-8d5a-349e2b43e9f3-kube-api-access-9cpms\") pod \"test-curl-8025-f5f4e6\" (UID: \"d77b337a-74a3-4c1f-8d5a-349e2b43e9f3\") " pod="opendatahub/test-curl-8025-f5f4e6" Sep 11 14:31:12.699172 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:12.698853 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8025-f5f4e6" Sep 11 14:31:13.005466 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:13.005430 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8025-f5f4e6"] Sep 11 14:31:13.007192 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:31:13.007158 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd77b337a_74a3_4c1f_8d5a_349e2b43e9f3.slice/crio-45060e770d9f5471a71ebd9c3801faa9c2bff63e30972cd03fbcec153d02178e WatchSource:0}: Error finding container 45060e770d9f5471a71ebd9c3801faa9c2bff63e30972cd03fbcec153d02178e: Status 404 returned error can't find the container with id 45060e770d9f5471a71ebd9c3801faa9c2bff63e30972cd03fbcec153d02178e Sep 11 14:31:13.453361 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:13.453094 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8025-952939" podUID="187b6e47-eb61-45a1-b254-58081f044712" containerName="test-curl-8025-952939" containerID="cri-o://abc4686f2559c909444172f55946c5578ec4b5273b3e96f5600e69ec34b3abfd" gracePeriod=2 Sep 11 14:31:13.454517 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:13.453009 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8025-f5f4e6" event={"ID":"d77b337a-74a3-4c1f-8d5a-349e2b43e9f3","Type":"ContainerStarted","Data":"66930146c3a3d058a29ebf69716573b213f24107c0f0275bf0e8ce4dcf330f71"} Sep 11 14:31:13.454632 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:13.454549 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8025-f5f4e6" event={"ID":"d77b337a-74a3-4c1f-8d5a-349e2b43e9f3","Type":"ContainerStarted","Data":"45060e770d9f5471a71ebd9c3801faa9c2bff63e30972cd03fbcec153d02178e"} Sep 11 14:31:13.455662 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:13.455627 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8025-952939\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="187b6e47-eb61-45a1-b254-58081f044712" pod="opendatahub/test-curl-8025-952939" Sep 11 14:31:13.483122 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:13.483071 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8025-f5f4e6" podStartSLOduration=1.483050851 podStartE2EDuration="1.483050851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:13.481066978 +0000 UTC m=+961.566523335" watchObservedRunningTime="2026-09-11 14:31:13.483050851 +0000 UTC m=+961.568507214" Sep 11 14:31:13.483834 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:13.483785 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8025-952939\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="187b6e47-eb61-45a1-b254-58081f044712" pod="opendatahub/test-curl-8025-952939" Sep 11 14:31:14.164510 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:14.164475 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8025-f5f4e6"] Sep 11 14:31:14.169252 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:14.169188 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8025-f5f4e6"] Sep 11 14:31:14.462981 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:14.462899 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8025-f5f4e6" container="test-curl-8025-f5f4e6" Sep 11 14:31:15.114899 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.114861 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn"] Sep 11 14:31:15.115460 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.115441 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d77b337a-74a3-4c1f-8d5a-349e2b43e9f3" containerName="test-curl-8025-f5f4e6" Sep 11 14:31:15.115460 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.115461 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d77b337a-74a3-4c1f-8d5a-349e2b43e9f3" containerName="test-curl-8025-f5f4e6" Sep 11 14:31:15.115615 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.115530 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d77b337a-74a3-4c1f-8d5a-349e2b43e9f3" containerName="test-curl-8025-f5f4e6" Sep 11 14:31:15.118835 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.118813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.122565 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.122538 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 14:31:15.122680 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.122632 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-84d3a8-openshift-default-dockercfg-hxlxk\"" Sep 11 14:31:15.140900 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.140874 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn"] Sep 11 14:31:15.168224 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.168190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-token\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.168648 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.168246 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-workload-certs\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.168648 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.168287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-envoy\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.168648 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.168366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-credential-socket\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.168648 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.168398 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcpl\" (UniqueName: \"kubernetes.io/projected/445d98e5-207b-43d2-b3e6-601ac81a3531-kube-api-access-tvcpl\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.168648 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.168506 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-podinfo\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.168648 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.168538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/445d98e5-207b-43d2-b3e6-601ac81a3531-istiod-ca-cert\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.168648 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.168566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-workload-socket\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.168936 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.168648 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-data\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.269281 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.269237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-podinfo\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.269281 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.269277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/445d98e5-207b-43d2-b3e6-601ac81a3531-istiod-ca-cert\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.269522 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.269300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-workload-socket\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.269522 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.269333 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-data\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.269522 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.269356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-token\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.269522 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.269392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-workload-certs\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.269522 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.269422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-envoy\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.269812 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.269554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-credential-socket\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.269812 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.269596 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcpl\" (UniqueName: \"kubernetes.io/projected/445d98e5-207b-43d2-b3e6-601ac81a3531-kube-api-access-tvcpl\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.269927 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.269906 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-data\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.269985 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.269957 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-workload-socket\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.270047 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.269997 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-workload-certs\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.270047 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.270021 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-credential-socket\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.270126 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.270063 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/445d98e5-207b-43d2-b3e6-601ac81a3531-istiod-ca-cert\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.271829 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.271802 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-podinfo\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.271944 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.271856 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-envoy\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.280752 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.280699 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-token\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.281084 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.281059 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcpl\" (UniqueName: \"kubernetes.io/projected/445d98e5-207b-43d2-b3e6-601ac81a3531-kube-api-access-tvcpl\") pod \"e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.337254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.337215 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:31:15.429008 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.428917 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:15.473813 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.473777 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" event={"ID":"8c26493f-6480-4559-a099-6090ac4830a0","Type":"ContainerStarted","Data":"276f60de0c46b4ec84dbe87bfcfd70516e4d90ae4762d290a95c9d34a25cf287"} Sep 11 14:31:15.473958 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.473928 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:15.474049 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.473995 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8025-f5f4e6" podUID="d77b337a-74a3-4c1f-8d5a-349e2b43e9f3" containerName="test-curl-8025-f5f4e6" containerID="cri-o://66930146c3a3d058a29ebf69716573b213f24107c0f0275bf0e8ce4dcf330f71" gracePeriod=2 Sep 11 14:31:15.524266 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.524210 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" podStartSLOduration=2.921609751 podStartE2EDuration="5.524187171s" totalImagesPullingTime="2.60257742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:15.52164849 +0000 UTC m=+963.607104855" watchObservedRunningTime="2026-09-11 14:31:15.524187171 +0000 UTC m=+963.609643532" Sep 11 14:31:15.667146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.667121 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8025-952939" Sep 11 14:31:15.669836 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:15.669805 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8025-952939\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="187b6e47-eb61-45a1-b254-58081f044712" pod="opendatahub/test-curl-8025-952939" Sep 11 14:31:15.741522 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.741495 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn"] Sep 11 14:31:15.743813 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:31:15.743781 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445d98e5_207b_43d2_b3e6_601ac81a3531.slice/crio-dfd9812eb3f2bcbcd69821a3a0cce24ee18ab7ad85c7929e095d02b9c30e1f38 WatchSource:0}: Error finding container dfd9812eb3f2bcbcd69821a3a0cce24ee18ab7ad85c7929e095d02b9c30e1f38: Status 404 returned error can't find the container with id dfd9812eb3f2bcbcd69821a3a0cce24ee18ab7ad85c7929e095d02b9c30e1f38 Sep 11 14:31:15.774538 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.774507 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqxww\" (UniqueName: \"kubernetes.io/projected/187b6e47-eb61-45a1-b254-58081f044712-kube-api-access-tqxww\") pod \"187b6e47-eb61-45a1-b254-58081f044712\" (UID: \"187b6e47-eb61-45a1-b254-58081f044712\") " Sep 11 14:31:15.776980 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.776948 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187b6e47-eb61-45a1-b254-58081f044712-kube-api-access-tqxww" (OuterVolumeSpecName: "kube-api-access-tqxww") pod "187b6e47-eb61-45a1-b254-58081f044712" (UID: "187b6e47-eb61-45a1-b254-58081f044712"). InnerVolumeSpecName "kube-api-access-tqxww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:31:15.876162 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:15.876125 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqxww\" (UniqueName: \"kubernetes.io/projected/187b6e47-eb61-45a1-b254-58081f044712-kube-api-access-tqxww\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:31:16.466967 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:16.466930 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="187b6e47-eb61-45a1-b254-58081f044712" path="/var/lib/kubelet/pods/187b6e47-eb61-45a1-b254-58081f044712/volumes" Sep 11 14:31:16.491874 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:16.491831 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" event={"ID":"445d98e5-207b-43d2-b3e6-601ac81a3531","Type":"ContainerStarted","Data":"dfd9812eb3f2bcbcd69821a3a0cce24ee18ab7ad85c7929e095d02b9c30e1f38"} Sep 11 14:31:16.493208 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:16.493173 2578 generic.go:356] "Generic (PLEG): container finished" podID="187b6e47-eb61-45a1-b254-58081f044712" containerID="abc4686f2559c909444172f55946c5578ec4b5273b3e96f5600e69ec34b3abfd" exitCode=137 Sep 11 14:31:16.493334 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:16.493229 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8025-952939" Sep 11 14:31:16.494070 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:16.493265 2578 scope.go:117] "RemoveContainer" containerID="abc4686f2559c909444172f55946c5578ec4b5273b3e96f5600e69ec34b3abfd" Sep 11 14:31:16.496129 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:16.496098 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8025-952939\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="187b6e47-eb61-45a1-b254-58081f044712" pod="opendatahub/test-curl-8025-952939" Sep 11 14:31:16.498501 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:16.498472 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8025-952939\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="187b6e47-eb61-45a1-b254-58081f044712" pod="opendatahub/test-curl-8025-952939" Sep 11 14:31:16.504391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:16.504369 2578 scope.go:117] "RemoveContainer" containerID="abc4686f2559c909444172f55946c5578ec4b5273b3e96f5600e69ec34b3abfd" Sep 11 14:31:16.504799 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:16.504716 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abc4686f2559c909444172f55946c5578ec4b5273b3e96f5600e69ec34b3abfd\": container with ID starting with abc4686f2559c909444172f55946c5578ec4b5273b3e96f5600e69ec34b3abfd not found: ID does not exist" containerID="abc4686f2559c909444172f55946c5578ec4b5273b3e96f5600e69ec34b3abfd" Sep 11 14:31:16.504905 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:16.504847 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc4686f2559c909444172f55946c5578ec4b5273b3e96f5600e69ec34b3abfd"} err="failed to get container status \"abc4686f2559c909444172f55946c5578ec4b5273b3e96f5600e69ec34b3abfd\": rpc error: code = NotFound desc = could not find container \"abc4686f2559c909444172f55946c5578ec4b5273b3e96f5600e69ec34b3abfd\": container with ID starting with abc4686f2559c909444172f55946c5578ec4b5273b3e96f5600e69ec34b3abfd not found: ID does not exist" Sep 11 14:31:18.214143 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.214107 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:31:18.214463 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.214189 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:31:18.214463 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.214226 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:31:18.300528 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.300504 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8025-f5f4e6" Sep 11 14:31:18.303347 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:18.303318 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8025-952939\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="187b6e47-eb61-45a1-b254-58081f044712" pod="opendatahub/test-curl-8025-952939" Sep 11 14:31:18.404482 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.404398 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cpms\" (UniqueName: \"kubernetes.io/projected/d77b337a-74a3-4c1f-8d5a-349e2b43e9f3-kube-api-access-9cpms\") pod \"d77b337a-74a3-4c1f-8d5a-349e2b43e9f3\" (UID: \"d77b337a-74a3-4c1f-8d5a-349e2b43e9f3\") " Sep 11 14:31:18.406368 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.406347 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77b337a-74a3-4c1f-8d5a-349e2b43e9f3-kube-api-access-9cpms" (OuterVolumeSpecName: "kube-api-access-9cpms") pod "d77b337a-74a3-4c1f-8d5a-349e2b43e9f3" (UID: "d77b337a-74a3-4c1f-8d5a-349e2b43e9f3"). InnerVolumeSpecName "kube-api-access-9cpms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:31:18.466129 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.466095 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d77b337a-74a3-4c1f-8d5a-349e2b43e9f3" path="/var/lib/kubelet/pods/d77b337a-74a3-4c1f-8d5a-349e2b43e9f3/volumes" Sep 11 14:31:18.505329 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.505291 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cpms\" (UniqueName: \"kubernetes.io/projected/d77b337a-74a3-4c1f-8d5a-349e2b43e9f3-kube-api-access-9cpms\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:31:18.509967 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.509931 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" event={"ID":"445d98e5-207b-43d2-b3e6-601ac81a3531","Type":"ContainerStarted","Data":"652444a18d9c854c87f89e485ebfd5981a9ac91feaccbf08fefd482c751dfc08"} Sep 11 14:31:18.511346 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.511318 2578 generic.go:356] "Generic (PLEG): container finished" podID="d77b337a-74a3-4c1f-8d5a-349e2b43e9f3" containerID="66930146c3a3d058a29ebf69716573b213f24107c0f0275bf0e8ce4dcf330f71" exitCode=137 Sep 11 14:31:18.511468 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.511358 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8025-f5f4e6" Sep 11 14:31:18.511468 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.511417 2578 scope.go:117] "RemoveContainer" containerID="66930146c3a3d058a29ebf69716573b213f24107c0f0275bf0e8ce4dcf330f71" Sep 11 14:31:18.520678 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.520661 2578 scope.go:117] "RemoveContainer" containerID="66930146c3a3d058a29ebf69716573b213f24107c0f0275bf0e8ce4dcf330f71" Sep 11 14:31:18.520993 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:18.520973 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66930146c3a3d058a29ebf69716573b213f24107c0f0275bf0e8ce4dcf330f71\": container with ID starting with 66930146c3a3d058a29ebf69716573b213f24107c0f0275bf0e8ce4dcf330f71 not found: ID does not exist" containerID="66930146c3a3d058a29ebf69716573b213f24107c0f0275bf0e8ce4dcf330f71" Sep 11 14:31:18.521060 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.521003 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66930146c3a3d058a29ebf69716573b213f24107c0f0275bf0e8ce4dcf330f71"} err="failed to get container status \"66930146c3a3d058a29ebf69716573b213f24107c0f0275bf0e8ce4dcf330f71\": rpc error: code = NotFound desc = could not find container \"66930146c3a3d058a29ebf69716573b213f24107c0f0275bf0e8ce4dcf330f71\": container with ID starting with 66930146c3a3d058a29ebf69716573b213f24107c0f0275bf0e8ce4dcf330f71 not found: ID does not exist" Sep 11 14:31:18.538304 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:18.538255 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" podStartSLOduration=1.070497221 podStartE2EDuration="3.538241583s" totalImagesPullingTime="2.467744362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:18.536259612 +0000 UTC m=+966.621716034" watchObservedRunningTime="2026-09-11 14:31:18.538241583 +0000 UTC m=+966.623697943" Sep 11 14:31:19.429745 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:19.429692 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:19.434700 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:19.434677 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:19.518795 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:19.518768 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:19.519666 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:19.519637 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:21.498562 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:21.498530 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:23.456102 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:23.456074 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" Sep 11 14:31:24.536844 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.536806 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw"] Sep 11 14:31:24.542077 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.542055 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" Sep 11 14:31:24.545783 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.545761 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-84d3a8\"" Sep 11 14:31:24.545969 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.545830 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-84d3a8-dockercfg-5rpwx\"" Sep 11 14:31:24.558099 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.558068 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw"] Sep 11 14:31:24.613640 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.613606 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm"] Sep 11 14:31:24.617340 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.617323 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" Sep 11 14:31:24.641687 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.641654 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm"] Sep 11 14:31:24.664160 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.664126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cl5m\" (UniqueName: \"kubernetes.io/projected/987bbcc0-f938-4055-9ce2-ee7077bd295f-kube-api-access-4cl5m\") pod \"payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw\" (UID: \"987bbcc0-f938-4055-9ce2-ee7077bd295f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" Sep 11 14:31:24.664390 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.664190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987bbcc0-f938-4055-9ce2-ee7077bd295f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw\" (UID: \"987bbcc0-f938-4055-9ce2-ee7077bd295f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" Sep 11 14:31:24.765026 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.764985 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cl5m\" (UniqueName: \"kubernetes.io/projected/987bbcc0-f938-4055-9ce2-ee7077bd295f-kube-api-access-4cl5m\") pod \"payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw\" (UID: \"987bbcc0-f938-4055-9ce2-ee7077bd295f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" Sep 11 14:31:24.765026 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.765030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/851c8b55-638e-4a59-ac94-4cd6959c151c-plugins-config-volume\") pod \"payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm\" (UID: \"851c8b55-638e-4a59-ac94-4cd6959c151c\") " pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" Sep 11 14:31:24.765227 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.765145 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tc9j\" (UniqueName: \"kubernetes.io/projected/851c8b55-638e-4a59-ac94-4cd6959c151c-kube-api-access-4tc9j\") pod \"payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm\" (UID: \"851c8b55-638e-4a59-ac94-4cd6959c151c\") " pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" Sep 11 14:31:24.765227 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.765199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987bbcc0-f938-4055-9ce2-ee7077bd295f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw\" (UID: \"987bbcc0-f938-4055-9ce2-ee7077bd295f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" Sep 11 14:31:24.765852 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.765833 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987bbcc0-f938-4055-9ce2-ee7077bd295f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw\" (UID: \"987bbcc0-f938-4055-9ce2-ee7077bd295f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" Sep 11 14:31:24.781501 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.781473 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cl5m\" (UniqueName: \"kubernetes.io/projected/987bbcc0-f938-4055-9ce2-ee7077bd295f-kube-api-access-4cl5m\") pod \"payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw\" (UID: \"987bbcc0-f938-4055-9ce2-ee7077bd295f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" Sep 11 14:31:24.853835 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.853793 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" Sep 11 14:31:24.866451 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.866416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/851c8b55-638e-4a59-ac94-4cd6959c151c-plugins-config-volume\") pod \"payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm\" (UID: \"851c8b55-638e-4a59-ac94-4cd6959c151c\") " pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" Sep 11 14:31:24.866563 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.866457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tc9j\" (UniqueName: \"kubernetes.io/projected/851c8b55-638e-4a59-ac94-4cd6959c151c-kube-api-access-4tc9j\") pod \"payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm\" (UID: \"851c8b55-638e-4a59-ac94-4cd6959c151c\") " pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" Sep 11 14:31:24.867218 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.867184 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/851c8b55-638e-4a59-ac94-4cd6959c151c-plugins-config-volume\") pod \"payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm\" (UID: \"851c8b55-638e-4a59-ac94-4cd6959c151c\") " pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" Sep 11 14:31:24.882854 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.882825 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tc9j\" (UniqueName: \"kubernetes.io/projected/851c8b55-638e-4a59-ac94-4cd6959c151c-kube-api-access-4tc9j\") pod \"payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm\" (UID: \"851c8b55-638e-4a59-ac94-4cd6959c151c\") " pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" Sep 11 14:31:24.927091 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:24.927036 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" Sep 11 14:31:25.135756 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:25.135718 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw"] Sep 11 14:31:25.138425 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:31:25.138397 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987bbcc0_f938_4055_9ce2_ee7077bd295f.slice/crio-e415469a4cafa82eddc6fd1f4b96ee47215758e5b1baf637316711b720a83793 WatchSource:0}: Error finding container e415469a4cafa82eddc6fd1f4b96ee47215758e5b1baf637316711b720a83793: Status 404 returned error can't find the container with id e415469a4cafa82eddc6fd1f4b96ee47215758e5b1baf637316711b720a83793 Sep 11 14:31:25.238084 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:25.238051 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm"] Sep 11 14:31:25.239492 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:31:25.239462 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851c8b55_638e_4a59_ac94_4cd6959c151c.slice/crio-131838564c5cfe661ea3d211699ed21b89faf31f3eaf00afaf187f7a8b514434 WatchSource:0}: Error finding container 131838564c5cfe661ea3d211699ed21b89faf31f3eaf00afaf187f7a8b514434: Status 404 returned error can't find the container with id 131838564c5cfe661ea3d211699ed21b89faf31f3eaf00afaf187f7a8b514434 Sep 11 14:31:25.569038 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:25.568940 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" event={"ID":"987bbcc0-f938-4055-9ce2-ee7077bd295f","Type":"ContainerStarted","Data":"bfea440793c56cb733624c10f2a8509bf511d7a3a8f59e6b5e7c4d6a9b6db6ec"} Sep 11 14:31:25.569038 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:25.569009 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" event={"ID":"987bbcc0-f938-4055-9ce2-ee7077bd295f","Type":"ContainerStarted","Data":"e415469a4cafa82eddc6fd1f4b96ee47215758e5b1baf637316711b720a83793"} Sep 11 14:31:25.569537 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:25.569238 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" Sep 11 14:31:25.571118 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:25.571095 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" event={"ID":"851c8b55-638e-4a59-ac94-4cd6959c151c","Type":"ContainerStarted","Data":"5c14e61134c2aa272aebd8ec44e64be19dfcc058e203dee6930a65c216b913ff"} Sep 11 14:31:25.571249 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:25.571136 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" Sep 11 14:31:25.571249 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:25.571153 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" event={"ID":"851c8b55-638e-4a59-ac94-4cd6959c151c","Type":"ContainerStarted","Data":"131838564c5cfe661ea3d211699ed21b89faf31f3eaf00afaf187f7a8b514434"} Sep 11 14:31:25.605914 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:25.605857 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" podStartSLOduration=1.6058393610000001 podStartE2EDuration="1.605839361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:25.60481599 +0000 UTC m=+973.690272348" watchObservedRunningTime="2026-09-11 14:31:25.605839361 +0000 UTC m=+973.691295721" Sep 11 14:31:25.640992 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:25.640932 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" podStartSLOduration=1.640910008 podStartE2EDuration="1.640910008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:25.63897155 +0000 UTC m=+973.724427931" watchObservedRunningTime="2026-09-11 14:31:25.640910008 +0000 UTC m=+973.726366366" Sep 11 14:31:32.530133 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:32.530095 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd"] Sep 11 14:31:32.535090 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:32.535070 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" Sep 11 14:31:32.537928 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:32.537902 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5a7035\"" Sep 11 14:31:32.538057 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:32.537901 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5a7035-dockercfg-hm5dr\"" Sep 11 14:31:32.559986 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:32.559955 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd"] Sep 11 14:31:32.646622 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:32.646592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbca1efd-7b92-4bba-8937-2aa60d7dc4e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd\" (UID: \"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" Sep 11 14:31:32.646810 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:32.646632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlshm\" (UniqueName: \"kubernetes.io/projected/cbca1efd-7b92-4bba-8937-2aa60d7dc4e8-kube-api-access-dlshm\") pod \"payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd\" (UID: \"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" Sep 11 14:31:32.747632 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:32.747592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbca1efd-7b92-4bba-8937-2aa60d7dc4e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd\" (UID: \"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" Sep 11 14:31:32.747632 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:32.747638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlshm\" (UniqueName: \"kubernetes.io/projected/cbca1efd-7b92-4bba-8937-2aa60d7dc4e8-kube-api-access-dlshm\") pod \"payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd\" (UID: \"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" Sep 11 14:31:32.748436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:32.748414 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbca1efd-7b92-4bba-8937-2aa60d7dc4e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd\" (UID: \"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" Sep 11 14:31:32.761188 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:32.761162 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlshm\" (UniqueName: \"kubernetes.io/projected/cbca1efd-7b92-4bba-8937-2aa60d7dc4e8-kube-api-access-dlshm\") pod \"payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd\" (UID: \"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" Sep 11 14:31:32.858462 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:32.858425 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" Sep 11 14:31:33.146614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:33.146583 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd"] Sep 11 14:31:33.147172 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:31:33.147143 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbca1efd_7b92_4bba_8937_2aa60d7dc4e8.slice/crio-3e57612a13b0f64ffbdc007458350e3428c54e860e3261adc8c26ff7bed956a0 WatchSource:0}: Error finding container 3e57612a13b0f64ffbdc007458350e3428c54e860e3261adc8c26ff7bed956a0: Status 404 returned error can't find the container with id 3e57612a13b0f64ffbdc007458350e3428c54e860e3261adc8c26ff7bed956a0 Sep 11 14:31:33.359703 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:33.359662 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:31:33.623036 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:33.623002 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" event={"ID":"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8","Type":"ContainerStarted","Data":"580799353aa1f3ccef77d480c37bc432230a5e8567afe4bc7c97e766d73ef6c1"} Sep 11 14:31:33.623420 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:33.623062 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" event={"ID":"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8","Type":"ContainerStarted","Data":"3e57612a13b0f64ffbdc007458350e3428c54e860e3261adc8c26ff7bed956a0"} Sep 11 14:31:33.623420 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:33.623112 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" Sep 11 14:31:33.649187 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:33.649122 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" podStartSLOduration=1.64910033 podStartE2EDuration="1.64910033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:33.647853589 +0000 UTC m=+981.733309948" watchObservedRunningTime="2026-09-11 14:31:33.64910033 +0000 UTC m=+981.734556691" Sep 11 14:31:36.578761 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:36.578653 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" Sep 11 14:31:36.579226 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:36.578833 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" Sep 11 14:31:38.015416 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.015375 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn"] Sep 11 14:31:38.015895 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.015626 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" podUID="8c26493f-6480-4559-a099-6090ac4830a0" containerName="maas-api" containerID="cri-o://276f60de0c46b4ec84dbe87bfcfd70516e4d90ae4762d290a95c9d34a25cf287" gracePeriod=30 Sep 11 14:31:38.211231 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.211189 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx"] Sep 11 14:31:38.211435 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.211405 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" podUID="ea6383be-e171-4a9f-b33a-c79d01d42026" containerName="payload-pre-processing" containerID="cri-o://65fa3132469dbbc51e4f1e3eb19b77c46da131bd9a0f3c4dd58648e8ed3ce48f" gracePeriod=30 Sep 11 14:31:38.369769 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.368344 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:38.512991 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.512960 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c26493f-6480-4559-a099-6090ac4830a0-openshift-service-ca\") pod \"8c26493f-6480-4559-a099-6090ac4830a0\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " Sep 11 14:31:38.513148 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.513016 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9bzp\" (UniqueName: \"kubernetes.io/projected/8c26493f-6480-4559-a099-6090ac4830a0-kube-api-access-w9bzp\") pod \"8c26493f-6480-4559-a099-6090ac4830a0\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " Sep 11 14:31:38.513148 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.513038 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-metrics-certs\") pod \"8c26493f-6480-4559-a099-6090ac4830a0\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " Sep 11 14:31:38.513148 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.513114 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-maas-api-tls\") pod \"8c26493f-6480-4559-a099-6090ac4830a0\" (UID: \"8c26493f-6480-4559-a099-6090ac4830a0\") " Sep 11 14:31:38.513747 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.513698 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c26493f-6480-4559-a099-6090ac4830a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c26493f-6480-4559-a099-6090ac4830a0" (UID: "8c26493f-6480-4559-a099-6090ac4830a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:31:38.515951 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.515895 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8c26493f-6480-4559-a099-6090ac4830a0" (UID: "8c26493f-6480-4559-a099-6090ac4830a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:31:38.516111 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.516085 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c26493f-6480-4559-a099-6090ac4830a0-kube-api-access-w9bzp" (OuterVolumeSpecName: "kube-api-access-w9bzp") pod "8c26493f-6480-4559-a099-6090ac4830a0" (UID: "8c26493f-6480-4559-a099-6090ac4830a0"). InnerVolumeSpecName "kube-api-access-w9bzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:31:38.516255 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.516231 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c26493f-6480-4559-a099-6090ac4830a0" (UID: "8c26493f-6480-4559-a099-6090ac4830a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:31:38.537621 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.537592 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" Sep 11 14:31:38.613900 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.613866 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:31:38.613900 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.613893 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c26493f-6480-4559-a099-6090ac4830a0-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:31:38.613900 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.613903 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9bzp\" (UniqueName: \"kubernetes.io/projected/8c26493f-6480-4559-a099-6090ac4830a0-kube-api-access-w9bzp\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:31:38.614131 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.613914 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c26493f-6480-4559-a099-6090ac4830a0-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:31:38.676754 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.675281 2578 generic.go:356] "Generic (PLEG): container finished" podID="ea6383be-e171-4a9f-b33a-c79d01d42026" containerID="65fa3132469dbbc51e4f1e3eb19b77c46da131bd9a0f3c4dd58648e8ed3ce48f" exitCode=0 Sep 11 14:31:38.676754 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.675458 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" event={"ID":"ea6383be-e171-4a9f-b33a-c79d01d42026","Type":"ContainerDied","Data":"65fa3132469dbbc51e4f1e3eb19b77c46da131bd9a0f3c4dd58648e8ed3ce48f"} Sep 11 14:31:38.676754 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.675526 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" event={"ID":"ea6383be-e171-4a9f-b33a-c79d01d42026","Type":"ContainerDied","Data":"debc9634a049c2b22eedebffb2351f341b672fde9c5c79a2281263b82ba42dd6"} Sep 11 14:31:38.676754 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.675573 2578 scope.go:117] "RemoveContainer" containerID="65fa3132469dbbc51e4f1e3eb19b77c46da131bd9a0f3c4dd58648e8ed3ce48f" Sep 11 14:31:38.676754 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.675815 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx" Sep 11 14:31:38.680211 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.680177 2578 generic.go:356] "Generic (PLEG): container finished" podID="8c26493f-6480-4559-a099-6090ac4830a0" containerID="276f60de0c46b4ec84dbe87bfcfd70516e4d90ae4762d290a95c9d34a25cf287" exitCode=0 Sep 11 14:31:38.680316 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.680278 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" Sep 11 14:31:38.680366 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.680322 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" event={"ID":"8c26493f-6480-4559-a099-6090ac4830a0","Type":"ContainerDied","Data":"276f60de0c46b4ec84dbe87bfcfd70516e4d90ae4762d290a95c9d34a25cf287"} Sep 11 14:31:38.680410 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.680384 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn" event={"ID":"8c26493f-6480-4559-a099-6090ac4830a0","Type":"ContainerDied","Data":"8220fe48221f06eb480217ef7a7317d11d55b34e8b2df7f9ce6fe2becfbe2719"} Sep 11 14:31:38.688231 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.688209 2578 scope.go:117] "RemoveContainer" containerID="65fa3132469dbbc51e4f1e3eb19b77c46da131bd9a0f3c4dd58648e8ed3ce48f" Sep 11 14:31:38.688532 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:38.688512 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fa3132469dbbc51e4f1e3eb19b77c46da131bd9a0f3c4dd58648e8ed3ce48f\": container with ID starting with 65fa3132469dbbc51e4f1e3eb19b77c46da131bd9a0f3c4dd58648e8ed3ce48f not found: ID does not exist" containerID="65fa3132469dbbc51e4f1e3eb19b77c46da131bd9a0f3c4dd58648e8ed3ce48f" Sep 11 14:31:38.688605 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.688542 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fa3132469dbbc51e4f1e3eb19b77c46da131bd9a0f3c4dd58648e8ed3ce48f"} err="failed to get container status \"65fa3132469dbbc51e4f1e3eb19b77c46da131bd9a0f3c4dd58648e8ed3ce48f\": rpc error: code = NotFound desc = could not find container \"65fa3132469dbbc51e4f1e3eb19b77c46da131bd9a0f3c4dd58648e8ed3ce48f\": container with ID starting with 65fa3132469dbbc51e4f1e3eb19b77c46da131bd9a0f3c4dd58648e8ed3ce48f not found: ID does not exist" Sep 11 14:31:38.688605 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.688559 2578 scope.go:117] "RemoveContainer" containerID="276f60de0c46b4ec84dbe87bfcfd70516e4d90ae4762d290a95c9d34a25cf287" Sep 11 14:31:38.697312 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.697293 2578 scope.go:117] "RemoveContainer" containerID="276f60de0c46b4ec84dbe87bfcfd70516e4d90ae4762d290a95c9d34a25cf287" Sep 11 14:31:38.697640 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:31:38.697604 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"276f60de0c46b4ec84dbe87bfcfd70516e4d90ae4762d290a95c9d34a25cf287\": container with ID starting with 276f60de0c46b4ec84dbe87bfcfd70516e4d90ae4762d290a95c9d34a25cf287 not found: ID does not exist" containerID="276f60de0c46b4ec84dbe87bfcfd70516e4d90ae4762d290a95c9d34a25cf287" Sep 11 14:31:38.697697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.697650 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276f60de0c46b4ec84dbe87bfcfd70516e4d90ae4762d290a95c9d34a25cf287"} err="failed to get container status \"276f60de0c46b4ec84dbe87bfcfd70516e4d90ae4762d290a95c9d34a25cf287\": rpc error: code = NotFound desc = could not find container \"276f60de0c46b4ec84dbe87bfcfd70516e4d90ae4762d290a95c9d34a25cf287\": container with ID starting with 276f60de0c46b4ec84dbe87bfcfd70516e4d90ae4762d290a95c9d34a25cf287 not found: ID does not exist" Sep 11 14:31:38.714714 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.714688 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jlhz\" (UniqueName: \"kubernetes.io/projected/ea6383be-e171-4a9f-b33a-c79d01d42026-kube-api-access-2jlhz\") pod \"ea6383be-e171-4a9f-b33a-c79d01d42026\" (UID: \"ea6383be-e171-4a9f-b33a-c79d01d42026\") " Sep 11 14:31:38.714864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.714782 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6383be-e171-4a9f-b33a-c79d01d42026-plugins-config-volume\") pod \"ea6383be-e171-4a9f-b33a-c79d01d42026\" (UID: \"ea6383be-e171-4a9f-b33a-c79d01d42026\") " Sep 11 14:31:38.715246 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.715222 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6383be-e171-4a9f-b33a-c79d01d42026-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea6383be-e171-4a9f-b33a-c79d01d42026" (UID: "ea6383be-e171-4a9f-b33a-c79d01d42026"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:31:38.716717 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.716694 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6383be-e171-4a9f-b33a-c79d01d42026-kube-api-access-2jlhz" (OuterVolumeSpecName: "kube-api-access-2jlhz") pod "ea6383be-e171-4a9f-b33a-c79d01d42026" (UID: "ea6383be-e171-4a9f-b33a-c79d01d42026"). InnerVolumeSpecName "kube-api-access-2jlhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:31:38.741202 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.741063 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn"] Sep 11 14:31:38.745477 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.745451 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b6b934c-769b47f558-5rwkn"] Sep 11 14:31:38.815693 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.815659 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jlhz\" (UniqueName: \"kubernetes.io/projected/ea6383be-e171-4a9f-b33a-c79d01d42026-kube-api-access-2jlhz\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:31:38.815693 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:38.815689 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6383be-e171-4a9f-b33a-c79d01d42026-plugins-config-volume\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:31:39.015298 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:39.015265 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx"] Sep 11 14:31:39.019676 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:39.019651 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b6b934c-5f66fd8c46-7kmbx"] Sep 11 14:31:40.465902 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:40.465871 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c26493f-6480-4559-a099-6090ac4830a0" path="/var/lib/kubelet/pods/8c26493f-6480-4559-a099-6090ac4830a0/volumes" Sep 11 14:31:40.466295 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:40.466279 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea6383be-e171-4a9f-b33a-c79d01d42026" path="/var/lib/kubelet/pods/ea6383be-e171-4a9f-b33a-c79d01d42026/volumes" Sep 11 14:31:43.723892 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:43.723797 2578 generic.go:356] "Generic (PLEG): container finished" podID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerID="652444a18d9c854c87f89e485ebfd5981a9ac91feaccbf08fefd482c751dfc08" exitCode=0 Sep 11 14:31:43.723892 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:43.723839 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" event={"ID":"445d98e5-207b-43d2-b3e6-601ac81a3531","Type":"ContainerDied","Data":"652444a18d9c854c87f89e485ebfd5981a9ac91feaccbf08fefd482c751dfc08"} Sep 11 14:31:43.724321 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:43.724237 2578 scope.go:117] "RemoveContainer" containerID="652444a18d9c854c87f89e485ebfd5981a9ac91feaccbf08fefd482c751dfc08" Sep 11 14:31:43.726181 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:43.726145 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:31:43.726277 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:43.726217 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:31:43.726277 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:43.726265 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:31:44.632826 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:44.632795 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" Sep 11 14:31:44.738750 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:44.738422 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" event={"ID":"445d98e5-207b-43d2-b3e6-601ac81a3531","Type":"ContainerStarted","Data":"51698a3feae626680ac305084c3105a865300bdb5b684948f3a72edddf61e06b"} Sep 11 14:31:45.429324 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:45.429284 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:45.430549 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:45.430525 2578 patch_prober.go:28] interesting pod/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:31:45.430626 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:45.430573 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" podUID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 14:31:46.209917 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.209872 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq"] Sep 11 14:31:46.210643 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.210612 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea6383be-e171-4a9f-b33a-c79d01d42026" containerName="payload-pre-processing" Sep 11 14:31:46.210643 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.210638 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea6383be-e171-4a9f-b33a-c79d01d42026" containerName="payload-pre-processing" Sep 11 14:31:46.210866 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.210653 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c26493f-6480-4559-a099-6090ac4830a0" containerName="maas-api" Sep 11 14:31:46.210866 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.210663 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c26493f-6480-4559-a099-6090ac4830a0" containerName="maas-api" Sep 11 14:31:46.210866 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.210779 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea6383be-e171-4a9f-b33a-c79d01d42026" containerName="payload-pre-processing" Sep 11 14:31:46.210866 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.210797 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c26493f-6480-4559-a099-6090ac4830a0" containerName="maas-api" Sep 11 14:31:46.218288 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.218264 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.222816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.222789 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 14:31:46.222816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.222810 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7cfns\"" Sep 11 14:31:46.223129 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.223102 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a6adb955-serving-cert\"" Sep 11 14:31:46.223241 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.223186 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:31:46.223445 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.223429 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 14:31:46.238000 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.237975 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq"] Sep 11 14:31:46.289073 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.289043 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-openshift-service-ca\") pod \"maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.289241 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.289080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjxq5\" (UniqueName: \"kubernetes.io/projected/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-kube-api-access-xjxq5\") pod \"maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.289241 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.289149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-metrics-certs\") pod \"maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.289241 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.289189 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-maas-api-tls\") pod \"maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.390434 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.390390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-metrics-certs\") pod \"maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.390614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.390457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-maas-api-tls\") pod \"maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.390614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.390513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-openshift-service-ca\") pod \"maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.390614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.390534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjxq5\" (UniqueName: \"kubernetes.io/projected/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-kube-api-access-xjxq5\") pod \"maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.391248 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.391213 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-openshift-service-ca\") pod \"maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.392980 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.392948 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-maas-api-tls\") pod \"maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.393087 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.392994 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-metrics-certs\") pod \"maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.404435 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.404414 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjxq5\" (UniqueName: \"kubernetes.io/projected/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-kube-api-access-xjxq5\") pod \"maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.429494 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.429472 2578 patch_prober.go:28] interesting pod/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:31:46.429588 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.429517 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" podUID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 14:31:46.530426 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.530348 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:46.851056 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:46.850851 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq"] Sep 11 14:31:46.851195 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:31:46.851117 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94c6cfc_ccbe_45e9_b8e2_91dce609ea74.slice/crio-ea9eed8c522032b4ce4f2ea10656a1930e9ec68807c4f235d88e94567ee681d3 WatchSource:0}: Error finding container ea9eed8c522032b4ce4f2ea10656a1930e9ec68807c4f235d88e94567ee681d3: Status 404 returned error can't find the container with id ea9eed8c522032b4ce4f2ea10656a1930e9ec68807c4f235d88e94567ee681d3 Sep 11 14:31:47.430100 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:47.430065 2578 patch_prober.go:28] interesting pod/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:31:47.430505 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:47.430120 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" podUID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 14:31:47.780516 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:47.780277 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" event={"ID":"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74","Type":"ContainerStarted","Data":"5fc9fba10978a0d6c85030671e4f67382a16b612b9477261f0ff4a5d0d47566a"} Sep 11 14:31:47.780516 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:47.780377 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" event={"ID":"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74","Type":"ContainerStarted","Data":"ea9eed8c522032b4ce4f2ea10656a1930e9ec68807c4f235d88e94567ee681d3"} Sep 11 14:31:47.780516 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:47.780433 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:47.827237 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:47.827184 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" podStartSLOduration=1.449789003 podStartE2EDuration="1.827168717s" totalImagesPullingTime="377.379714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:47.82404562 +0000 UTC m=+995.909501993" watchObservedRunningTime="2026-09-11 14:31:47.827168717 +0000 UTC m=+995.912625416" Sep 11 14:31:48.430363 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:48.430325 2578 patch_prober.go:28] interesting pod/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:31:48.430765 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:48.430405 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" podUID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 14:31:49.429413 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:49.429373 2578 patch_prober.go:28] interesting pod/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:31:49.429608 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:49.429443 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" podUID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 14:31:49.519491 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:49.519451 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:50.433927 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:50.433897 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:50.435085 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:50.435062 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:31:53.792531 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:53.792503 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:31:56.743238 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:31:56.743203 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:32:09.347686 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.347652 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq"] Sep 11 14:32:09.348064 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.347877 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" podUID="a94c6cfc-ccbe-45e9-b8e2-91dce609ea74" containerName="maas-api" containerID="cri-o://5fc9fba10978a0d6c85030671e4f67382a16b612b9477261f0ff4a5d0d47566a" gracePeriod=30 Sep 11 14:32:09.613493 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.613469 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:32:09.710853 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.710821 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-maas-api-tls\") pod \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " Sep 11 14:32:09.711010 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.710866 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-openshift-service-ca\") pod \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " Sep 11 14:32:09.711010 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.710918 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjxq5\" (UniqueName: \"kubernetes.io/projected/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-kube-api-access-xjxq5\") pod \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " Sep 11 14:32:09.711010 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.710946 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-metrics-certs\") pod \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\" (UID: \"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74\") " Sep 11 14:32:09.711249 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.711220 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a94c6cfc-ccbe-45e9-b8e2-91dce609ea74" (UID: "a94c6cfc-ccbe-45e9-b8e2-91dce609ea74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:32:09.712942 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.712918 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a94c6cfc-ccbe-45e9-b8e2-91dce609ea74" (UID: "a94c6cfc-ccbe-45e9-b8e2-91dce609ea74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:32:09.713017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.712987 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a94c6cfc-ccbe-45e9-b8e2-91dce609ea74" (UID: "a94c6cfc-ccbe-45e9-b8e2-91dce609ea74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:32:09.713075 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.713053 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-kube-api-access-xjxq5" (OuterVolumeSpecName: "kube-api-access-xjxq5") pod "a94c6cfc-ccbe-45e9-b8e2-91dce609ea74" (UID: "a94c6cfc-ccbe-45e9-b8e2-91dce609ea74"). InnerVolumeSpecName "kube-api-access-xjxq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:09.811530 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.811499 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:32:09.811530 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.811524 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:32:09.811530 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.811536 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjxq5\" (UniqueName: \"kubernetes.io/projected/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-kube-api-access-xjxq5\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:32:09.811786 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.811544 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:32:09.920108 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.920026 2578 generic.go:356] "Generic (PLEG): container finished" podID="a94c6cfc-ccbe-45e9-b8e2-91dce609ea74" containerID="5fc9fba10978a0d6c85030671e4f67382a16b612b9477261f0ff4a5d0d47566a" exitCode=0 Sep 11 14:32:09.920108 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.920075 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" event={"ID":"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74","Type":"ContainerDied","Data":"5fc9fba10978a0d6c85030671e4f67382a16b612b9477261f0ff4a5d0d47566a"} Sep 11 14:32:09.920108 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.920090 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" Sep 11 14:32:09.920386 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.920136 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq" event={"ID":"a94c6cfc-ccbe-45e9-b8e2-91dce609ea74","Type":"ContainerDied","Data":"ea9eed8c522032b4ce4f2ea10656a1930e9ec68807c4f235d88e94567ee681d3"} Sep 11 14:32:09.920386 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.920157 2578 scope.go:117] "RemoveContainer" containerID="5fc9fba10978a0d6c85030671e4f67382a16b612b9477261f0ff4a5d0d47566a" Sep 11 14:32:09.930553 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.930535 2578 scope.go:117] "RemoveContainer" containerID="5fc9fba10978a0d6c85030671e4f67382a16b612b9477261f0ff4a5d0d47566a" Sep 11 14:32:09.930812 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:32:09.930792 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc9fba10978a0d6c85030671e4f67382a16b612b9477261f0ff4a5d0d47566a\": container with ID starting with 5fc9fba10978a0d6c85030671e4f67382a16b612b9477261f0ff4a5d0d47566a not found: ID does not exist" containerID="5fc9fba10978a0d6c85030671e4f67382a16b612b9477261f0ff4a5d0d47566a" Sep 11 14:32:09.930867 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.930821 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc9fba10978a0d6c85030671e4f67382a16b612b9477261f0ff4a5d0d47566a"} err="failed to get container status \"5fc9fba10978a0d6c85030671e4f67382a16b612b9477261f0ff4a5d0d47566a\": rpc error: code = NotFound desc = could not find container \"5fc9fba10978a0d6c85030671e4f67382a16b612b9477261f0ff4a5d0d47566a\": container with ID starting with 5fc9fba10978a0d6c85030671e4f67382a16b612b9477261f0ff4a5d0d47566a not found: ID does not exist" Sep 11 14:32:09.965365 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.965340 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq"] Sep 11 14:32:09.970709 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:09.970686 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a6adb955-7b668c569d-vx5kq"] Sep 11 14:32:10.464715 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:10.464681 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a94c6cfc-ccbe-45e9-b8e2-91dce609ea74" path="/var/lib/kubelet/pods/a94c6cfc-ccbe-45e9-b8e2-91dce609ea74/volumes" Sep 11 14:32:16.152673 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.152636 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:32:16.834914 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.834880 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp"] Sep 11 14:32:16.835401 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.835382 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94c6cfc-ccbe-45e9-b8e2-91dce609ea74" containerName="maas-api" Sep 11 14:32:16.835499 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.835404 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94c6cfc-ccbe-45e9-b8e2-91dce609ea74" containerName="maas-api" Sep 11 14:32:16.835555 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.835501 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94c6cfc-ccbe-45e9-b8e2-91dce609ea74" containerName="maas-api" Sep 11 14:32:16.840252 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.840228 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:16.844307 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.844285 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7cfns\"" Sep 11 14:32:16.845375 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.845357 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 14:32:16.845686 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.845661 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:32:16.845686 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.845672 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 14:32:16.845686 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.845651 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-61c24488-serving-cert\"" Sep 11 14:32:16.879942 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.879921 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp"] Sep 11 14:32:16.982583 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.982552 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e65a2422-ab53-4be1-b2e6-808b4042c2be-metrics-certs\") pod \"maas-api-e2e-ait-61c24488-b757569b8-tpdlp\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:16.982772 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.982607 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65a2422-ab53-4be1-b2e6-808b4042c2be-maas-api-tls\") pod \"maas-api-e2e-ait-61c24488-b757569b8-tpdlp\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:16.982772 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.982665 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxvbm\" (UniqueName: \"kubernetes.io/projected/e65a2422-ab53-4be1-b2e6-808b4042c2be-kube-api-access-cxvbm\") pod \"maas-api-e2e-ait-61c24488-b757569b8-tpdlp\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:16.982772 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:16.982753 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65a2422-ab53-4be1-b2e6-808b4042c2be-openshift-service-ca\") pod \"maas-api-e2e-ait-61c24488-b757569b8-tpdlp\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:17.083798 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:17.083764 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65a2422-ab53-4be1-b2e6-808b4042c2be-openshift-service-ca\") pod \"maas-api-e2e-ait-61c24488-b757569b8-tpdlp\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:17.084001 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:17.083836 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e65a2422-ab53-4be1-b2e6-808b4042c2be-metrics-certs\") pod \"maas-api-e2e-ait-61c24488-b757569b8-tpdlp\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:17.084001 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:17.083893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65a2422-ab53-4be1-b2e6-808b4042c2be-maas-api-tls\") pod \"maas-api-e2e-ait-61c24488-b757569b8-tpdlp\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:17.084001 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:17.083912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxvbm\" (UniqueName: \"kubernetes.io/projected/e65a2422-ab53-4be1-b2e6-808b4042c2be-kube-api-access-cxvbm\") pod \"maas-api-e2e-ait-61c24488-b757569b8-tpdlp\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:17.084491 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:17.084461 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65a2422-ab53-4be1-b2e6-808b4042c2be-openshift-service-ca\") pod \"maas-api-e2e-ait-61c24488-b757569b8-tpdlp\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:17.086273 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:17.086220 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e65a2422-ab53-4be1-b2e6-808b4042c2be-metrics-certs\") pod \"maas-api-e2e-ait-61c24488-b757569b8-tpdlp\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:17.086373 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:17.086272 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65a2422-ab53-4be1-b2e6-808b4042c2be-maas-api-tls\") pod \"maas-api-e2e-ait-61c24488-b757569b8-tpdlp\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:17.112303 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:17.112279 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxvbm\" (UniqueName: \"kubernetes.io/projected/e65a2422-ab53-4be1-b2e6-808b4042c2be-kube-api-access-cxvbm\") pod \"maas-api-e2e-ait-61c24488-b757569b8-tpdlp\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:17.152701 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:17.152673 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:17.440843 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:17.440820 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp"] Sep 11 14:32:17.441780 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:32:17.441755 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65a2422_ab53_4be1_b2e6_808b4042c2be.slice/crio-95877bdab38f4f7ef4266566a64fb8b607802c4e86ce842e6599510df2c0a5f2 WatchSource:0}: Error finding container 95877bdab38f4f7ef4266566a64fb8b607802c4e86ce842e6599510df2c0a5f2: Status 404 returned error can't find the container with id 95877bdab38f4f7ef4266566a64fb8b607802c4e86ce842e6599510df2c0a5f2 Sep 11 14:32:17.974932 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:17.974896 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" event={"ID":"e65a2422-ab53-4be1-b2e6-808b4042c2be","Type":"ContainerStarted","Data":"71e2a3b4dcb997ec1c31d6723a3a2ba98871e7ee944b16423b350d83f297f6ff"} Sep 11 14:32:17.975106 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:17.974944 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" event={"ID":"e65a2422-ab53-4be1-b2e6-808b4042c2be","Type":"ContainerStarted","Data":"95877bdab38f4f7ef4266566a64fb8b607802c4e86ce842e6599510df2c0a5f2"} Sep 11 14:32:17.975106 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:17.975028 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:23.987096 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:23.987068 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:24.019970 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:24.019917 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" podStartSLOduration=7.600103807 podStartE2EDuration="8.019903709s" totalImagesPullingTime="419.799902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:18.011010864 +0000 UTC m=+1026.096467225" watchObservedRunningTime="2026-09-11 14:32:24.019903709 +0000 UTC m=+1032.105360068" Sep 11 14:32:30.748596 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:30.748563 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:32:48.042293 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:48.042256 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:32:57.396339 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.396298 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp"] Sep 11 14:32:57.396852 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.396609 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" podUID="e65a2422-ab53-4be1-b2e6-808b4042c2be" containerName="maas-api" containerID="cri-o://71e2a3b4dcb997ec1c31d6723a3a2ba98871e7ee944b16423b350d83f297f6ff" gracePeriod=30 Sep 11 14:32:57.717127 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.717098 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:57.862365 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.862332 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65a2422-ab53-4be1-b2e6-808b4042c2be-maas-api-tls\") pod \"e65a2422-ab53-4be1-b2e6-808b4042c2be\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " Sep 11 14:32:57.862365 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.862369 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65a2422-ab53-4be1-b2e6-808b4042c2be-openshift-service-ca\") pod \"e65a2422-ab53-4be1-b2e6-808b4042c2be\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " Sep 11 14:32:57.862571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.862500 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e65a2422-ab53-4be1-b2e6-808b4042c2be-metrics-certs\") pod \"e65a2422-ab53-4be1-b2e6-808b4042c2be\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " Sep 11 14:32:57.862571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.862538 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxvbm\" (UniqueName: \"kubernetes.io/projected/e65a2422-ab53-4be1-b2e6-808b4042c2be-kube-api-access-cxvbm\") pod \"e65a2422-ab53-4be1-b2e6-808b4042c2be\" (UID: \"e65a2422-ab53-4be1-b2e6-808b4042c2be\") " Sep 11 14:32:57.862778 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.862753 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65a2422-ab53-4be1-b2e6-808b4042c2be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e65a2422-ab53-4be1-b2e6-808b4042c2be" (UID: "e65a2422-ab53-4be1-b2e6-808b4042c2be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:32:57.862913 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.862900 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65a2422-ab53-4be1-b2e6-808b4042c2be-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:32:57.864516 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.864477 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e65a2422-ab53-4be1-b2e6-808b4042c2be-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e65a2422-ab53-4be1-b2e6-808b4042c2be" (UID: "e65a2422-ab53-4be1-b2e6-808b4042c2be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:32:57.864516 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.864503 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e65a2422-ab53-4be1-b2e6-808b4042c2be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e65a2422-ab53-4be1-b2e6-808b4042c2be" (UID: "e65a2422-ab53-4be1-b2e6-808b4042c2be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:32:57.864660 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.864570 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65a2422-ab53-4be1-b2e6-808b4042c2be-kube-api-access-cxvbm" (OuterVolumeSpecName: "kube-api-access-cxvbm") pod "e65a2422-ab53-4be1-b2e6-808b4042c2be" (UID: "e65a2422-ab53-4be1-b2e6-808b4042c2be"). InnerVolumeSpecName "kube-api-access-cxvbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:57.963795 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.963702 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65a2422-ab53-4be1-b2e6-808b4042c2be-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:32:57.963795 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.963749 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e65a2422-ab53-4be1-b2e6-808b4042c2be-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:32:57.963795 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:57.963760 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxvbm\" (UniqueName: \"kubernetes.io/projected/e65a2422-ab53-4be1-b2e6-808b4042c2be-kube-api-access-cxvbm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:32:58.219157 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:58.219070 2578 generic.go:356] "Generic (PLEG): container finished" podID="e65a2422-ab53-4be1-b2e6-808b4042c2be" containerID="71e2a3b4dcb997ec1c31d6723a3a2ba98871e7ee944b16423b350d83f297f6ff" exitCode=0 Sep 11 14:32:58.219157 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:58.219114 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" Sep 11 14:32:58.219352 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:58.219151 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" event={"ID":"e65a2422-ab53-4be1-b2e6-808b4042c2be","Type":"ContainerDied","Data":"71e2a3b4dcb997ec1c31d6723a3a2ba98871e7ee944b16423b350d83f297f6ff"} Sep 11 14:32:58.219352 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:58.219233 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp" event={"ID":"e65a2422-ab53-4be1-b2e6-808b4042c2be","Type":"ContainerDied","Data":"95877bdab38f4f7ef4266566a64fb8b607802c4e86ce842e6599510df2c0a5f2"} Sep 11 14:32:58.219352 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:58.219256 2578 scope.go:117] "RemoveContainer" containerID="71e2a3b4dcb997ec1c31d6723a3a2ba98871e7ee944b16423b350d83f297f6ff" Sep 11 14:32:58.229277 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:58.229257 2578 scope.go:117] "RemoveContainer" containerID="71e2a3b4dcb997ec1c31d6723a3a2ba98871e7ee944b16423b350d83f297f6ff" Sep 11 14:32:58.229575 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:32:58.229548 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e2a3b4dcb997ec1c31d6723a3a2ba98871e7ee944b16423b350d83f297f6ff\": container with ID starting with 71e2a3b4dcb997ec1c31d6723a3a2ba98871e7ee944b16423b350d83f297f6ff not found: ID does not exist" containerID="71e2a3b4dcb997ec1c31d6723a3a2ba98871e7ee944b16423b350d83f297f6ff" Sep 11 14:32:58.229633 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:58.229582 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e2a3b4dcb997ec1c31d6723a3a2ba98871e7ee944b16423b350d83f297f6ff"} err="failed to get container status \"71e2a3b4dcb997ec1c31d6723a3a2ba98871e7ee944b16423b350d83f297f6ff\": rpc error: code = NotFound desc = could not find container \"71e2a3b4dcb997ec1c31d6723a3a2ba98871e7ee944b16423b350d83f297f6ff\": container with ID starting with 71e2a3b4dcb997ec1c31d6723a3a2ba98871e7ee944b16423b350d83f297f6ff not found: ID does not exist" Sep 11 14:32:58.259149 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:58.259118 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp"] Sep 11 14:32:58.261674 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:58.261652 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61c24488-b757569b8-tpdlp"] Sep 11 14:32:58.465662 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:32:58.465625 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e65a2422-ab53-4be1-b2e6-808b4042c2be" path="/var/lib/kubelet/pods/e65a2422-ab53-4be1-b2e6-808b4042c2be/volumes" Sep 11 14:33:00.120420 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.120391 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w"] Sep 11 14:33:00.120834 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.120820 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e65a2422-ab53-4be1-b2e6-808b4042c2be" containerName="maas-api" Sep 11 14:33:00.120889 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.120836 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e65a2422-ab53-4be1-b2e6-808b4042c2be" containerName="maas-api" Sep 11 14:33:00.120929 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.120916 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e65a2422-ab53-4be1-b2e6-808b4042c2be" containerName="maas-api" Sep 11 14:33:00.125260 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.125240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.128235 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.128212 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-93c38c4c-openshift-default-dockercfg-m6tdk\"" Sep 11 14:33:00.151111 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.151086 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w"] Sep 11 14:33:00.185686 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.185659 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-token\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.185818 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.185696 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-envoy\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.185818 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.185715 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-workload-socket\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.185921 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.185838 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-data\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.185921 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.185896 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-workload-certs\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.185921 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.185917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf5lf\" (UniqueName: \"kubernetes.io/projected/2f02337c-6171-493e-a6f7-54e8ea0e908e-kube-api-access-bf5lf\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.186037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.185947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-podinfo\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.186037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.185965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-credential-socket\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.186037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.186022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f02337c-6171-493e-a6f7-54e8ea0e908e-istiod-ca-cert\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.286458 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.286427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-data\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.286636 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.286482 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-workload-certs\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.286636 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.286500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf5lf\" (UniqueName: \"kubernetes.io/projected/2f02337c-6171-493e-a6f7-54e8ea0e908e-kube-api-access-bf5lf\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.286636 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.286536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-podinfo\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.286636 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.286556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-credential-socket\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.286636 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.286577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f02337c-6171-493e-a6f7-54e8ea0e908e-istiod-ca-cert\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.286933 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.286634 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-token\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.286933 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.286677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-envoy\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.286933 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.286713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-workload-socket\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.287563 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.286952 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-data\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.287563 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.287073 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-credential-socket\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.287563 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.287162 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-workload-socket\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.287796 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.287775 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-workload-certs\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.287872 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.287776 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f02337c-6171-493e-a6f7-54e8ea0e908e-istiod-ca-cert\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.289124 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.289106 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-envoy\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.289300 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.289279 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-podinfo\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.303910 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.303884 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-token\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.304149 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.304131 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf5lf\" (UniqueName: \"kubernetes.io/projected/2f02337c-6171-493e-a6f7-54e8ea0e908e-kube-api-access-bf5lf\") pod \"e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.435617 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.435517 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:00.730205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.730180 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w"] Sep 11 14:33:00.734234 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:33:00.734199 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f02337c_6171_493e_a6f7_54e8ea0e908e.slice/crio-01e429a348a4fd12164263b6ad871d5ffea8107f772918635c38113ca563bfd6 WatchSource:0}: Error finding container 01e429a348a4fd12164263b6ad871d5ffea8107f772918635c38113ca563bfd6: Status 404 returned error can't find the container with id 01e429a348a4fd12164263b6ad871d5ffea8107f772918635c38113ca563bfd6 Sep 11 14:33:00.736248 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.736214 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:33:00.736350 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.736294 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:33:00.736350 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:00.736335 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:33:01.241274 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:01.241246 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" event={"ID":"2f02337c-6171-493e-a6f7-54e8ea0e908e","Type":"ContainerStarted","Data":"46c48ae71a60806790cfddb333aeb018a2a49f94f66a5d5cfdfcff697901ce17"} Sep 11 14:33:01.241674 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:01.241294 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" event={"ID":"2f02337c-6171-493e-a6f7-54e8ea0e908e","Type":"ContainerStarted","Data":"01e429a348a4fd12164263b6ad871d5ffea8107f772918635c38113ca563bfd6"} Sep 11 14:33:01.281067 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:01.281015 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" podStartSLOduration=1.280998275 podStartE2EDuration="1.280998275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:01.277439277 +0000 UTC m=+1069.362895639" watchObservedRunningTime="2026-09-11 14:33:01.280998275 +0000 UTC m=+1069.366454634" Sep 11 14:33:01.435813 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:01.435779 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:01.440486 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:01.440463 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:02.247461 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:02.247426 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:02.248472 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:02.248446 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:04.525273 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.525239 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7"] Sep 11 14:33:04.528949 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.528932 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:04.534041 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.534010 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-93c38c4c-serving-cert\"" Sep 11 14:33:04.534162 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.534021 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 14:33:04.535282 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.535261 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7cfns\"" Sep 11 14:33:04.535406 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.535303 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 14:33:04.535864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.535847 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:33:04.558441 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.558401 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7"] Sep 11 14:33:04.630973 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.630941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e723e9a-2423-4117-b016-44b6469b4785-openshift-service-ca\") pod \"maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:04.631169 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.631009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-maas-api-tls\") pod \"maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:04.631169 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.631037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-metrics-certs\") pod \"maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:04.631169 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.631059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxslq\" (UniqueName: \"kubernetes.io/projected/8e723e9a-2423-4117-b016-44b6469b4785-kube-api-access-qxslq\") pod \"maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:04.731909 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.731875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxslq\" (UniqueName: \"kubernetes.io/projected/8e723e9a-2423-4117-b016-44b6469b4785-kube-api-access-qxslq\") pod \"maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:04.732085 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.731991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e723e9a-2423-4117-b016-44b6469b4785-openshift-service-ca\") pod \"maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:04.732085 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.732034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-maas-api-tls\") pod \"maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:04.732085 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.732056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-metrics-certs\") pod \"maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:04.732286 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:33:04.732153 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-serving-cert: secret "maas-api-e2e-derive-93c38c4c-serving-cert" not found Sep 11 14:33:04.732286 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:33:04.732234 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-maas-api-tls podName:8e723e9a-2423-4117-b016-44b6469b4785 nodeName:}" failed. No retries permitted until 2026-09-11 14:33:05.232215946 +0000 UTC m=+1073.317672308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-maas-api-tls") pod "maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" (UID: "8e723e9a-2423-4117-b016-44b6469b4785") : secret "maas-api-e2e-derive-93c38c4c-serving-cert" not found Sep 11 14:33:04.732756 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.732713 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e723e9a-2423-4117-b016-44b6469b4785-openshift-service-ca\") pod \"maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:04.734445 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.734428 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-metrics-certs\") pod \"maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:04.749230 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:04.749197 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxslq\" (UniqueName: \"kubernetes.io/projected/8e723e9a-2423-4117-b016-44b6469b4785-kube-api-access-qxslq\") pod \"maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:05.237607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:05.237526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-maas-api-tls\") pod \"maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:05.239884 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:05.239861 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-maas-api-tls\") pod \"maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:05.439964 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:05.439929 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:05.738025 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:05.738002 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7"] Sep 11 14:33:05.739182 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:33:05.739157 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e723e9a_2423_4117_b016_44b6469b4785.slice/crio-605c522b8270608cfabe630c5e458401f11877c67ecaf079ca6d727616e6038c WatchSource:0}: Error finding container 605c522b8270608cfabe630c5e458401f11877c67ecaf079ca6d727616e6038c: Status 404 returned error can't find the container with id 605c522b8270608cfabe630c5e458401f11877c67ecaf079ca6d727616e6038c Sep 11 14:33:06.271603 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:06.271564 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" event={"ID":"8e723e9a-2423-4117-b016-44b6469b4785","Type":"ContainerStarted","Data":"81de7670a54c5ffdead0f72fa86d370ebae45b3c392f897742e1b15122b62493"} Sep 11 14:33:06.271799 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:06.271624 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" event={"ID":"8e723e9a-2423-4117-b016-44b6469b4785","Type":"ContainerStarted","Data":"605c522b8270608cfabe630c5e458401f11877c67ecaf079ca6d727616e6038c"} Sep 11 14:33:06.271799 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:06.271652 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:06.300243 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:06.300171 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" podStartSLOduration=1.973001778 podStartE2EDuration="2.300152314s" totalImagesPullingTime="327.150536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:06.298512866 +0000 UTC m=+1074.383969226" watchObservedRunningTime="2026-09-11 14:33:06.300152314 +0000 UTC m=+1074.385608675" Sep 11 14:33:07.245404 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:07.245362 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:33:12.282853 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:12.282826 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:25.677523 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:25.677487 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:33:27.416219 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:27.416186 2578 generic.go:356] "Generic (PLEG): container finished" podID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerID="46c48ae71a60806790cfddb333aeb018a2a49f94f66a5d5cfdfcff697901ce17" exitCode=0 Sep 11 14:33:27.416769 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:27.416243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" event={"ID":"2f02337c-6171-493e-a6f7-54e8ea0e908e","Type":"ContainerDied","Data":"46c48ae71a60806790cfddb333aeb018a2a49f94f66a5d5cfdfcff697901ce17"} Sep 11 14:33:27.416769 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:27.416616 2578 scope.go:117] "RemoveContainer" containerID="46c48ae71a60806790cfddb333aeb018a2a49f94f66a5d5cfdfcff697901ce17" Sep 11 14:33:27.417775 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:27.417744 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:33:27.417859 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:27.417814 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:33:27.417859 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:27.417849 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:33:28.424011 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:28.423977 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" event={"ID":"2f02337c-6171-493e-a6f7-54e8ea0e908e","Type":"ContainerStarted","Data":"5f732f7d51f698c0e5f6ebf4d5bd1a3101bc63b69eec466661a472a1bd54ec51"} Sep 11 14:33:28.436299 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:28.436276 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:28.437471 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:28.437449 2578 patch_prober.go:28] interesting pod/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:33:28.437527 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:28.437505 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 11 14:33:29.436582 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:29.436545 2578 patch_prober.go:28] interesting pod/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:33:29.437004 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:29.436597 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 11 14:33:30.439642 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:30.439608 2578 patch_prober.go:28] interesting pod/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:33:30.440111 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:30.439662 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 11 14:33:31.436374 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:31.436337 2578 patch_prober.go:28] interesting pod/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:33:31.436546 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:31.436398 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 11 14:33:32.247485 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:32.247450 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:32.436053 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:32.436018 2578 patch_prober.go:28] interesting pod/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:33:32.436219 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:32.436099 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 11 14:33:33.436695 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:33.436659 2578 patch_prober.go:28] interesting pod/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:33:33.437094 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:33.436721 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 11 14:33:34.436687 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:34.436650 2578 patch_prober.go:28] interesting pod/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:33:34.436881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:34.436716 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 11 14:33:35.439382 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:35.439350 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:35.440312 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:35.440294 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:40.981026 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:40.980994 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm"] Sep 11 14:33:40.981403 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:40.981202 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" podUID="851c8b55-638e-4a59-ac94-4cd6959c151c" containerName="payload-processing" containerID="cri-o://5c14e61134c2aa272aebd8ec44e64be19dfcc058e203dee6930a65c216b913ff" gracePeriod=30 Sep 11 14:33:40.998635 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:40.998596 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw"] Sep 11 14:33:40.998882 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:40.998851 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" podUID="987bbcc0-f938-4055-9ce2-ee7077bd295f" containerName="payload-pre-processing" containerID="cri-o://bfea440793c56cb733624c10f2a8509bf511d7a3a8f59e6b5e7c4d6a9b6db6ec" gracePeriod=30 Sep 11 14:33:41.249743 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.249701 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" Sep 11 14:33:41.287092 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.286583 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" Sep 11 14:33:41.391438 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.391407 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/851c8b55-638e-4a59-ac94-4cd6959c151c-plugins-config-volume\") pod \"851c8b55-638e-4a59-ac94-4cd6959c151c\" (UID: \"851c8b55-638e-4a59-ac94-4cd6959c151c\") " Sep 11 14:33:41.391627 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.391444 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tc9j\" (UniqueName: \"kubernetes.io/projected/851c8b55-638e-4a59-ac94-4cd6959c151c-kube-api-access-4tc9j\") pod \"851c8b55-638e-4a59-ac94-4cd6959c151c\" (UID: \"851c8b55-638e-4a59-ac94-4cd6959c151c\") " Sep 11 14:33:41.391627 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.391529 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cl5m\" (UniqueName: \"kubernetes.io/projected/987bbcc0-f938-4055-9ce2-ee7077bd295f-kube-api-access-4cl5m\") pod \"987bbcc0-f938-4055-9ce2-ee7077bd295f\" (UID: \"987bbcc0-f938-4055-9ce2-ee7077bd295f\") " Sep 11 14:33:41.391627 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.391574 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987bbcc0-f938-4055-9ce2-ee7077bd295f-plugins-config-volume\") pod \"987bbcc0-f938-4055-9ce2-ee7077bd295f\" (UID: \"987bbcc0-f938-4055-9ce2-ee7077bd295f\") " Sep 11 14:33:41.391985 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.391959 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851c8b55-638e-4a59-ac94-4cd6959c151c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "851c8b55-638e-4a59-ac94-4cd6959c151c" (UID: "851c8b55-638e-4a59-ac94-4cd6959c151c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:33:41.392050 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.392005 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987bbcc0-f938-4055-9ce2-ee7077bd295f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "987bbcc0-f938-4055-9ce2-ee7077bd295f" (UID: "987bbcc0-f938-4055-9ce2-ee7077bd295f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:33:41.393544 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.393527 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851c8b55-638e-4a59-ac94-4cd6959c151c-kube-api-access-4tc9j" (OuterVolumeSpecName: "kube-api-access-4tc9j") pod "851c8b55-638e-4a59-ac94-4cd6959c151c" (UID: "851c8b55-638e-4a59-ac94-4cd6959c151c"). InnerVolumeSpecName "kube-api-access-4tc9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:41.393616 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.393596 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987bbcc0-f938-4055-9ce2-ee7077bd295f-kube-api-access-4cl5m" (OuterVolumeSpecName: "kube-api-access-4cl5m") pod "987bbcc0-f938-4055-9ce2-ee7077bd295f" (UID: "987bbcc0-f938-4055-9ce2-ee7077bd295f"). InnerVolumeSpecName "kube-api-access-4cl5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:41.492777 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.492683 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/851c8b55-638e-4a59-ac94-4cd6959c151c-plugins-config-volume\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:41.492777 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.492708 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tc9j\" (UniqueName: \"kubernetes.io/projected/851c8b55-638e-4a59-ac94-4cd6959c151c-kube-api-access-4tc9j\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:41.492777 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.492718 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cl5m\" (UniqueName: \"kubernetes.io/projected/987bbcc0-f938-4055-9ce2-ee7077bd295f-kube-api-access-4cl5m\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:41.492777 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.492755 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987bbcc0-f938-4055-9ce2-ee7077bd295f-plugins-config-volume\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:41.521881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.521849 2578 generic.go:356] "Generic (PLEG): container finished" podID="987bbcc0-f938-4055-9ce2-ee7077bd295f" containerID="bfea440793c56cb733624c10f2a8509bf511d7a3a8f59e6b5e7c4d6a9b6db6ec" exitCode=0 Sep 11 14:33:41.522016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.521896 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" Sep 11 14:33:41.522016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.521935 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" event={"ID":"987bbcc0-f938-4055-9ce2-ee7077bd295f","Type":"ContainerDied","Data":"bfea440793c56cb733624c10f2a8509bf511d7a3a8f59e6b5e7c4d6a9b6db6ec"} Sep 11 14:33:41.522016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.521992 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw" event={"ID":"987bbcc0-f938-4055-9ce2-ee7077bd295f","Type":"ContainerDied","Data":"e415469a4cafa82eddc6fd1f4b96ee47215758e5b1baf637316711b720a83793"} Sep 11 14:33:41.522163 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.522020 2578 scope.go:117] "RemoveContainer" containerID="bfea440793c56cb733624c10f2a8509bf511d7a3a8f59e6b5e7c4d6a9b6db6ec" Sep 11 14:33:41.523455 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.523426 2578 generic.go:356] "Generic (PLEG): container finished" podID="851c8b55-638e-4a59-ac94-4cd6959c151c" containerID="5c14e61134c2aa272aebd8ec44e64be19dfcc058e203dee6930a65c216b913ff" exitCode=0 Sep 11 14:33:41.523571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.523465 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" Sep 11 14:33:41.523571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.523500 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" event={"ID":"851c8b55-638e-4a59-ac94-4cd6959c151c","Type":"ContainerDied","Data":"5c14e61134c2aa272aebd8ec44e64be19dfcc058e203dee6930a65c216b913ff"} Sep 11 14:33:41.523571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.523552 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm" event={"ID":"851c8b55-638e-4a59-ac94-4cd6959c151c","Type":"ContainerDied","Data":"131838564c5cfe661ea3d211699ed21b89faf31f3eaf00afaf187f7a8b514434"} Sep 11 14:33:41.537798 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.537643 2578 scope.go:117] "RemoveContainer" containerID="bfea440793c56cb733624c10f2a8509bf511d7a3a8f59e6b5e7c4d6a9b6db6ec" Sep 11 14:33:41.538326 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:33:41.538128 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfea440793c56cb733624c10f2a8509bf511d7a3a8f59e6b5e7c4d6a9b6db6ec\": container with ID starting with bfea440793c56cb733624c10f2a8509bf511d7a3a8f59e6b5e7c4d6a9b6db6ec not found: ID does not exist" containerID="bfea440793c56cb733624c10f2a8509bf511d7a3a8f59e6b5e7c4d6a9b6db6ec" Sep 11 14:33:41.538326 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.538163 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfea440793c56cb733624c10f2a8509bf511d7a3a8f59e6b5e7c4d6a9b6db6ec"} err="failed to get container status \"bfea440793c56cb733624c10f2a8509bf511d7a3a8f59e6b5e7c4d6a9b6db6ec\": rpc error: code = NotFound desc = could not find container \"bfea440793c56cb733624c10f2a8509bf511d7a3a8f59e6b5e7c4d6a9b6db6ec\": container with ID starting with bfea440793c56cb733624c10f2a8509bf511d7a3a8f59e6b5e7c4d6a9b6db6ec not found: ID does not exist" Sep 11 14:33:41.538326 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.538184 2578 scope.go:117] "RemoveContainer" containerID="5c14e61134c2aa272aebd8ec44e64be19dfcc058e203dee6930a65c216b913ff" Sep 11 14:33:41.546882 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.546860 2578 scope.go:117] "RemoveContainer" containerID="5c14e61134c2aa272aebd8ec44e64be19dfcc058e203dee6930a65c216b913ff" Sep 11 14:33:41.547131 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:33:41.547114 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c14e61134c2aa272aebd8ec44e64be19dfcc058e203dee6930a65c216b913ff\": container with ID starting with 5c14e61134c2aa272aebd8ec44e64be19dfcc058e203dee6930a65c216b913ff not found: ID does not exist" containerID="5c14e61134c2aa272aebd8ec44e64be19dfcc058e203dee6930a65c216b913ff" Sep 11 14:33:41.547194 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.547137 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c14e61134c2aa272aebd8ec44e64be19dfcc058e203dee6930a65c216b913ff"} err="failed to get container status \"5c14e61134c2aa272aebd8ec44e64be19dfcc058e203dee6930a65c216b913ff\": rpc error: code = NotFound desc = could not find container \"5c14e61134c2aa272aebd8ec44e64be19dfcc058e203dee6930a65c216b913ff\": container with ID starting with 5c14e61134c2aa272aebd8ec44e64be19dfcc058e203dee6930a65c216b913ff not found: ID does not exist" Sep 11 14:33:41.572016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.571993 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm"] Sep 11 14:33:41.576968 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.576946 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-84d3a8-967787f5c-tt6cm"] Sep 11 14:33:41.604009 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.603987 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw"] Sep 11 14:33:41.607675 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:41.607655 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-84d3a8-6b65bc9894-rlddw"] Sep 11 14:33:42.464678 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:42.464639 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="851c8b55-638e-4a59-ac94-4cd6959c151c" path="/var/lib/kubelet/pods/851c8b55-638e-4a59-ac94-4cd6959c151c/volumes" Sep 11 14:33:42.465087 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:42.465021 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987bbcc0-f938-4055-9ce2-ee7077bd295f" path="/var/lib/kubelet/pods/987bbcc0-f938-4055-9ce2-ee7077bd295f/volumes" Sep 11 14:33:47.265543 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.264889 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7"] Sep 11 14:33:47.265543 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.265189 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" podUID="8e723e9a-2423-4117-b016-44b6469b4785" containerName="maas-api" containerID="cri-o://81de7670a54c5ffdead0f72fa86d370ebae45b3c392f897742e1b15122b62493" gracePeriod=30 Sep 11 14:33:47.279388 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.279333 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" podUID="8e723e9a-2423-4117-b016-44b6469b4785" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.69:8443/health\": dial tcp 10.134.0.69:8443: connect: connection refused" Sep 11 14:33:47.573766 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.572526 2578 generic.go:356] "Generic (PLEG): container finished" podID="8e723e9a-2423-4117-b016-44b6469b4785" containerID="81de7670a54c5ffdead0f72fa86d370ebae45b3c392f897742e1b15122b62493" exitCode=0 Sep 11 14:33:47.573766 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.572647 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" event={"ID":"8e723e9a-2423-4117-b016-44b6469b4785","Type":"ContainerDied","Data":"81de7670a54c5ffdead0f72fa86d370ebae45b3c392f897742e1b15122b62493"} Sep 11 14:33:47.573766 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.572705 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" event={"ID":"8e723e9a-2423-4117-b016-44b6469b4785","Type":"ContainerDied","Data":"605c522b8270608cfabe630c5e458401f11877c67ecaf079ca6d727616e6038c"} Sep 11 14:33:47.573766 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.572749 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="605c522b8270608cfabe630c5e458401f11877c67ecaf079ca6d727616e6038c" Sep 11 14:33:47.589710 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.589685 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:47.649630 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.649593 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e723e9a-2423-4117-b016-44b6469b4785-openshift-service-ca\") pod \"8e723e9a-2423-4117-b016-44b6469b4785\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " Sep 11 14:33:47.649845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.649640 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-maas-api-tls\") pod \"8e723e9a-2423-4117-b016-44b6469b4785\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " Sep 11 14:33:47.649845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.649782 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-metrics-certs\") pod \"8e723e9a-2423-4117-b016-44b6469b4785\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " Sep 11 14:33:47.649971 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.649884 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxslq\" (UniqueName: \"kubernetes.io/projected/8e723e9a-2423-4117-b016-44b6469b4785-kube-api-access-qxslq\") pod \"8e723e9a-2423-4117-b016-44b6469b4785\" (UID: \"8e723e9a-2423-4117-b016-44b6469b4785\") " Sep 11 14:33:47.652715 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.652686 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e723e9a-2423-4117-b016-44b6469b4785-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e723e9a-2423-4117-b016-44b6469b4785" (UID: "8e723e9a-2423-4117-b016-44b6469b4785"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:33:47.654896 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.654869 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8e723e9a-2423-4117-b016-44b6469b4785" (UID: "8e723e9a-2423-4117-b016-44b6469b4785"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:33:47.655181 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.655141 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e723e9a-2423-4117-b016-44b6469b4785-kube-api-access-qxslq" (OuterVolumeSpecName: "kube-api-access-qxslq") pod "8e723e9a-2423-4117-b016-44b6469b4785" (UID: "8e723e9a-2423-4117-b016-44b6469b4785"). InnerVolumeSpecName "kube-api-access-qxslq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:47.655909 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.655573 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e723e9a-2423-4117-b016-44b6469b4785" (UID: "8e723e9a-2423-4117-b016-44b6469b4785"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:33:47.751794 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.751762 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:47.751794 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.751791 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxslq\" (UniqueName: \"kubernetes.io/projected/8e723e9a-2423-4117-b016-44b6469b4785-kube-api-access-qxslq\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:47.751794 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.751800 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e723e9a-2423-4117-b016-44b6469b4785-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:47.752081 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:47.751809 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e723e9a-2423-4117-b016-44b6469b4785-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:48.116705 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:48.116664 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w"] Sep 11 14:33:48.117348 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:48.117290 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" containerID="cri-o://5f732f7d51f698c0e5f6ebf4d5bd1a3101bc63b69eec466661a472a1bd54ec51" gracePeriod=30 Sep 11 14:33:48.450903 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:48.450817 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:33:48.588002 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:48.587968 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7" Sep 11 14:33:48.616029 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:48.615990 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7"] Sep 11 14:33:48.621879 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:48.621853 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-93c38c4c-c6f569b6f-fz6c7"] Sep 11 14:33:50.445033 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:50.444983 2578 patch_prober.go:28] interesting pod/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:33:50.445487 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:50.445069 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:33:50.464721 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:50.464687 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e723e9a-2423-4117-b016-44b6469b4785" path="/var/lib/kubelet/pods/8e723e9a-2423-4117-b016-44b6469b4785/volumes" Sep 11 14:33:52.873088 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:52.873003 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn"] Sep 11 14:33:52.873523 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:52.873397 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" podUID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerName="istio-proxy" containerID="cri-o://51698a3feae626680ac305084c3105a865300bdb5b684948f3a72edddf61e06b" gracePeriod=30 Sep 11 14:33:53.375584 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.375560 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:53.403023 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.402961 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-envoy\") pod \"2f02337c-6171-493e-a6f7-54e8ea0e908e\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " Sep 11 14:33:53.403023 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.402990 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-credential-socket\") pod \"2f02337c-6171-493e-a6f7-54e8ea0e908e\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " Sep 11 14:33:53.403190 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403046 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-token\") pod \"2f02337c-6171-493e-a6f7-54e8ea0e908e\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " Sep 11 14:33:53.403190 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403061 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-workload-certs\") pod \"2f02337c-6171-493e-a6f7-54e8ea0e908e\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " Sep 11 14:33:53.403190 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403084 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-podinfo\") pod \"2f02337c-6171-493e-a6f7-54e8ea0e908e\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " Sep 11 14:33:53.403190 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403112 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-workload-socket\") pod \"2f02337c-6171-493e-a6f7-54e8ea0e908e\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " Sep 11 14:33:53.403190 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403154 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf5lf\" (UniqueName: \"kubernetes.io/projected/2f02337c-6171-493e-a6f7-54e8ea0e908e-kube-api-access-bf5lf\") pod \"2f02337c-6171-493e-a6f7-54e8ea0e908e\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " Sep 11 14:33:53.403437 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403219 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-data\") pod \"2f02337c-6171-493e-a6f7-54e8ea0e908e\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " Sep 11 14:33:53.403437 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403260 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f02337c-6171-493e-a6f7-54e8ea0e908e-istiod-ca-cert\") pod \"2f02337c-6171-493e-a6f7-54e8ea0e908e\" (UID: \"2f02337c-6171-493e-a6f7-54e8ea0e908e\") " Sep 11 14:33:53.403437 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403388 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2f02337c-6171-493e-a6f7-54e8ea0e908e" (UID: "2f02337c-6171-493e-a6f7-54e8ea0e908e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:53.403437 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403421 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2f02337c-6171-493e-a6f7-54e8ea0e908e" (UID: "2f02337c-6171-493e-a6f7-54e8ea0e908e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:53.403437 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403403 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2f02337c-6171-493e-a6f7-54e8ea0e908e" (UID: "2f02337c-6171-493e-a6f7-54e8ea0e908e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:53.403790 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403612 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-credential-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:53.403790 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403640 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-workload-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:53.403790 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403656 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-workload-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:53.403790 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403615 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-data" (OuterVolumeSpecName: "istio-data") pod "2f02337c-6171-493e-a6f7-54e8ea0e908e" (UID: "2f02337c-6171-493e-a6f7-54e8ea0e908e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:53.403790 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.403675 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f02337c-6171-493e-a6f7-54e8ea0e908e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2f02337c-6171-493e-a6f7-54e8ea0e908e" (UID: "2f02337c-6171-493e-a6f7-54e8ea0e908e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:33:53.405962 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.405938 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2f02337c-6171-493e-a6f7-54e8ea0e908e" (UID: "2f02337c-6171-493e-a6f7-54e8ea0e908e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:53.406154 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.405966 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-token" (OuterVolumeSpecName: "istio-token") pod "2f02337c-6171-493e-a6f7-54e8ea0e908e" (UID: "2f02337c-6171-493e-a6f7-54e8ea0e908e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:53.406314 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.405431 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2f02337c-6171-493e-a6f7-54e8ea0e908e" (UID: "2f02337c-6171-493e-a6f7-54e8ea0e908e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:33:53.406482 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.406460 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f02337c-6171-493e-a6f7-54e8ea0e908e-kube-api-access-bf5lf" (OuterVolumeSpecName: "kube-api-access-bf5lf") pod "2f02337c-6171-493e-a6f7-54e8ea0e908e" (UID: "2f02337c-6171-493e-a6f7-54e8ea0e908e"). InnerVolumeSpecName "kube-api-access-bf5lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:53.505076 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.505043 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-data\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:53.505076 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.505070 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f02337c-6171-493e-a6f7-54e8ea0e908e-istiod-ca-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:53.505076 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.505080 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-envoy\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:53.505299 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.505088 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-token\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:53.505299 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.505097 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f02337c-6171-493e-a6f7-54e8ea0e908e-istio-podinfo\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:53.505299 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.505105 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf5lf\" (UniqueName: \"kubernetes.io/projected/2f02337c-6171-493e-a6f7-54e8ea0e908e-kube-api-access-bf5lf\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:53.621279 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.621241 2578 generic.go:356] "Generic (PLEG): container finished" podID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerID="5f732f7d51f698c0e5f6ebf4d5bd1a3101bc63b69eec466661a472a1bd54ec51" exitCode=0 Sep 11 14:33:53.621451 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.621288 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" Sep 11 14:33:53.621451 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.621316 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" event={"ID":"2f02337c-6171-493e-a6f7-54e8ea0e908e","Type":"ContainerDied","Data":"5f732f7d51f698c0e5f6ebf4d5bd1a3101bc63b69eec466661a472a1bd54ec51"} Sep 11 14:33:53.621451 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.621365 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w" event={"ID":"2f02337c-6171-493e-a6f7-54e8ea0e908e","Type":"ContainerDied","Data":"01e429a348a4fd12164263b6ad871d5ffea8107f772918635c38113ca563bfd6"} Sep 11 14:33:53.621451 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.621389 2578 scope.go:117] "RemoveContainer" containerID="5f732f7d51f698c0e5f6ebf4d5bd1a3101bc63b69eec466661a472a1bd54ec51" Sep 11 14:33:53.632172 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.632153 2578 scope.go:117] "RemoveContainer" containerID="46c48ae71a60806790cfddb333aeb018a2a49f94f66a5d5cfdfcff697901ce17" Sep 11 14:33:53.640475 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.640457 2578 scope.go:117] "RemoveContainer" containerID="5f732f7d51f698c0e5f6ebf4d5bd1a3101bc63b69eec466661a472a1bd54ec51" Sep 11 14:33:53.640817 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:33:53.640792 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f732f7d51f698c0e5f6ebf4d5bd1a3101bc63b69eec466661a472a1bd54ec51\": container with ID starting with 5f732f7d51f698c0e5f6ebf4d5bd1a3101bc63b69eec466661a472a1bd54ec51 not found: ID does not exist" containerID="5f732f7d51f698c0e5f6ebf4d5bd1a3101bc63b69eec466661a472a1bd54ec51" Sep 11 14:33:53.640901 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.640824 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f732f7d51f698c0e5f6ebf4d5bd1a3101bc63b69eec466661a472a1bd54ec51"} err="failed to get container status \"5f732f7d51f698c0e5f6ebf4d5bd1a3101bc63b69eec466661a472a1bd54ec51\": rpc error: code = NotFound desc = could not find container \"5f732f7d51f698c0e5f6ebf4d5bd1a3101bc63b69eec466661a472a1bd54ec51\": container with ID starting with 5f732f7d51f698c0e5f6ebf4d5bd1a3101bc63b69eec466661a472a1bd54ec51 not found: ID does not exist" Sep 11 14:33:53.640901 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.640839 2578 scope.go:117] "RemoveContainer" containerID="46c48ae71a60806790cfddb333aeb018a2a49f94f66a5d5cfdfcff697901ce17" Sep 11 14:33:53.641097 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:33:53.641079 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c48ae71a60806790cfddb333aeb018a2a49f94f66a5d5cfdfcff697901ce17\": container with ID starting with 46c48ae71a60806790cfddb333aeb018a2a49f94f66a5d5cfdfcff697901ce17 not found: ID does not exist" containerID="46c48ae71a60806790cfddb333aeb018a2a49f94f66a5d5cfdfcff697901ce17" Sep 11 14:33:53.641143 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.641103 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c48ae71a60806790cfddb333aeb018a2a49f94f66a5d5cfdfcff697901ce17"} err="failed to get container status \"46c48ae71a60806790cfddb333aeb018a2a49f94f66a5d5cfdfcff697901ce17\": rpc error: code = NotFound desc = could not find container \"46c48ae71a60806790cfddb333aeb018a2a49f94f66a5d5cfdfcff697901ce17\": container with ID starting with 46c48ae71a60806790cfddb333aeb018a2a49f94f66a5d5cfdfcff697901ce17 not found: ID does not exist" Sep 11 14:33:53.660889 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.660817 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w"] Sep 11 14:33:53.674163 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.674131 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-93c38c4c-openshift-default-6f7496c7c6-nmm5w"] Sep 11 14:33:53.801934 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.801903 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx"] Sep 11 14:33:53.802394 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802377 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" Sep 11 14:33:53.802497 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802397 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" Sep 11 14:33:53.802497 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802411 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" Sep 11 14:33:53.802497 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802419 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" Sep 11 14:33:53.802497 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802438 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851c8b55-638e-4a59-ac94-4cd6959c151c" containerName="payload-processing" Sep 11 14:33:53.802497 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802447 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="851c8b55-638e-4a59-ac94-4cd6959c151c" containerName="payload-processing" Sep 11 14:33:53.802497 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802469 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e723e9a-2423-4117-b016-44b6469b4785" containerName="maas-api" Sep 11 14:33:53.802497 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802477 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e723e9a-2423-4117-b016-44b6469b4785" containerName="maas-api" Sep 11 14:33:53.802864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802502 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987bbcc0-f938-4055-9ce2-ee7077bd295f" containerName="payload-pre-processing" Sep 11 14:33:53.802864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802512 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="987bbcc0-f938-4055-9ce2-ee7077bd295f" containerName="payload-pre-processing" Sep 11 14:33:53.802864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802615 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e723e9a-2423-4117-b016-44b6469b4785" containerName="maas-api" Sep 11 14:33:53.802864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802630 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" Sep 11 14:33:53.802864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802642 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="851c8b55-638e-4a59-ac94-4cd6959c151c" containerName="payload-processing" Sep 11 14:33:53.802864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802652 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="987bbcc0-f938-4055-9ce2-ee7077bd295f" containerName="payload-pre-processing" Sep 11 14:33:53.802864 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.802663 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" containerName="istio-proxy" Sep 11 14:33:53.807759 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.807701 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:53.810683 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.810661 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-259c79-openshift-default-dockercfg-dl77d\"" Sep 11 14:33:53.825646 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.825622 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx"] Sep 11 14:33:53.909327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.909294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-token\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:53.909759 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.909331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-workload-socket\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:53.909759 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.909359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-workload-certs\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:53.909759 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.909379 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-podinfo\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:53.909759 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.909443 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-data\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:53.909759 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.909507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-credential-socket\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:53.909759 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.909530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-envoy\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:53.909759 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.909556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfns2\" (UniqueName: \"kubernetes.io/projected/a02a1fa4-087c-49a8-a89f-f0017b3528f7-kube-api-access-xfns2\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:53.909759 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:53.909618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istiod-ca-cert\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.010404 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.010321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-data\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.010404 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.010386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-credential-socket\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.010623 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.010414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-envoy\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.010623 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.010449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfns2\" (UniqueName: \"kubernetes.io/projected/a02a1fa4-087c-49a8-a89f-f0017b3528f7-kube-api-access-xfns2\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.010623 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.010488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istiod-ca-cert\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.010623 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.010539 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-token\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.010623 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.010569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-workload-socket\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.010623 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.010613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-workload-certs\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.011000 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.010638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-podinfo\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.011000 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.010804 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-data\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.011000 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.010850 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-credential-socket\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.011124 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.011002 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-workload-socket\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.011124 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.011080 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-workload-certs\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.011250 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.011230 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istiod-ca-cert\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.012678 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.012659 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-envoy\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.012995 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.012978 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-podinfo\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.020836 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.020815 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-token\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.021578 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.021562 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfns2\" (UniqueName: \"kubernetes.io/projected/a02a1fa4-087c-49a8-a89f-f0017b3528f7-kube-api-access-xfns2\") pod \"e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.117968 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.117932 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:54.404358 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.404332 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx"] Sep 11 14:33:54.405825 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:33:54.405794 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02a1fa4_087c_49a8_a89f_f0017b3528f7.slice/crio-1d5e7fcee9def3ec330399a76dce89f79cce6515d629b8aeddc13d268b4bf769 WatchSource:0}: Error finding container 1d5e7fcee9def3ec330399a76dce89f79cce6515d629b8aeddc13d268b4bf769: Status 404 returned error can't find the container with id 1d5e7fcee9def3ec330399a76dce89f79cce6515d629b8aeddc13d268b4bf769 Sep 11 14:33:54.408012 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.407980 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:33:54.408091 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.408047 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:33:54.408132 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.408093 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:33:54.464738 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.464696 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f02337c-6171-493e-a6f7-54e8ea0e908e" path="/var/lib/kubelet/pods/2f02337c-6171-493e-a6f7-54e8ea0e908e/volumes" Sep 11 14:33:54.631780 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.631714 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" event={"ID":"a02a1fa4-087c-49a8-a89f-f0017b3528f7","Type":"ContainerStarted","Data":"9d62ba5905706a4fec07287d9b5be6b2b05d075decd0b0eb2fa3f5d3b1956e64"} Sep 11 14:33:54.631969 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.631804 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" event={"ID":"a02a1fa4-087c-49a8-a89f-f0017b3528f7","Type":"ContainerStarted","Data":"1d5e7fcee9def3ec330399a76dce89f79cce6515d629b8aeddc13d268b4bf769"} Sep 11 14:33:54.660684 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:54.660578 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" podStartSLOduration=1.660555105 podStartE2EDuration="1.660555105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:54.659313424 +0000 UTC m=+1122.744769801" watchObservedRunningTime="2026-09-11 14:33:54.660555105 +0000 UTC m=+1122.746011468" Sep 11 14:33:55.118647 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:55.118610 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:55.123678 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:55.123646 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:55.639219 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:55.639182 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:55.640329 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:55.640301 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:33:58.133826 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.133792 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:33:58.249298 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.249200 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-podinfo\") pod \"445d98e5-207b-43d2-b3e6-601ac81a3531\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " Sep 11 14:33:58.249298 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.249246 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-token\") pod \"445d98e5-207b-43d2-b3e6-601ac81a3531\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " Sep 11 14:33:58.249298 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.249269 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-data\") pod \"445d98e5-207b-43d2-b3e6-601ac81a3531\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " Sep 11 14:33:58.249596 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.249338 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-workload-socket\") pod \"445d98e5-207b-43d2-b3e6-601ac81a3531\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " Sep 11 14:33:58.249596 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.249367 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvcpl\" (UniqueName: \"kubernetes.io/projected/445d98e5-207b-43d2-b3e6-601ac81a3531-kube-api-access-tvcpl\") pod \"445d98e5-207b-43d2-b3e6-601ac81a3531\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " Sep 11 14:33:58.249596 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.249400 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-envoy\") pod \"445d98e5-207b-43d2-b3e6-601ac81a3531\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " Sep 11 14:33:58.249596 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.249424 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-credential-socket\") pod \"445d98e5-207b-43d2-b3e6-601ac81a3531\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " Sep 11 14:33:58.249596 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.249461 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/445d98e5-207b-43d2-b3e6-601ac81a3531-istiod-ca-cert\") pod \"445d98e5-207b-43d2-b3e6-601ac81a3531\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " Sep 11 14:33:58.249596 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.249506 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-workload-certs\") pod \"445d98e5-207b-43d2-b3e6-601ac81a3531\" (UID: \"445d98e5-207b-43d2-b3e6-601ac81a3531\") " Sep 11 14:33:58.249956 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.249907 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "445d98e5-207b-43d2-b3e6-601ac81a3531" (UID: "445d98e5-207b-43d2-b3e6-601ac81a3531"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:58.250019 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.249950 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "445d98e5-207b-43d2-b3e6-601ac81a3531" (UID: "445d98e5-207b-43d2-b3e6-601ac81a3531"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:58.250253 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.250229 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445d98e5-207b-43d2-b3e6-601ac81a3531-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "445d98e5-207b-43d2-b3e6-601ac81a3531" (UID: "445d98e5-207b-43d2-b3e6-601ac81a3531"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:33:58.250486 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.250454 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-data" (OuterVolumeSpecName: "istio-data") pod "445d98e5-207b-43d2-b3e6-601ac81a3531" (UID: "445d98e5-207b-43d2-b3e6-601ac81a3531"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:58.251500 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.250678 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-workload-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:58.251500 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.250701 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-data\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:58.251500 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.250716 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-credential-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:58.251500 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.250750 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/445d98e5-207b-43d2-b3e6-601ac81a3531-istiod-ca-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:58.252010 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.251991 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "445d98e5-207b-43d2-b3e6-601ac81a3531" (UID: "445d98e5-207b-43d2-b3e6-601ac81a3531"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:58.252515 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.252478 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "445d98e5-207b-43d2-b3e6-601ac81a3531" (UID: "445d98e5-207b-43d2-b3e6-601ac81a3531"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:33:58.252915 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.252879 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-token" (OuterVolumeSpecName: "istio-token") pod "445d98e5-207b-43d2-b3e6-601ac81a3531" (UID: "445d98e5-207b-43d2-b3e6-601ac81a3531"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:58.253106 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.253083 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "445d98e5-207b-43d2-b3e6-601ac81a3531" (UID: "445d98e5-207b-43d2-b3e6-601ac81a3531"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:58.253996 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.253970 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445d98e5-207b-43d2-b3e6-601ac81a3531-kube-api-access-tvcpl" (OuterVolumeSpecName: "kube-api-access-tvcpl") pod "445d98e5-207b-43d2-b3e6-601ac81a3531" (UID: "445d98e5-207b-43d2-b3e6-601ac81a3531"). InnerVolumeSpecName "kube-api-access-tvcpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:58.351982 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.351947 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-workload-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:58.351982 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.351979 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvcpl\" (UniqueName: \"kubernetes.io/projected/445d98e5-207b-43d2-b3e6-601ac81a3531-kube-api-access-tvcpl\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:58.351982 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.351990 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-envoy\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:58.351982 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.351998 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-podinfo\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:58.352253 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.352007 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/445d98e5-207b-43d2-b3e6-601ac81a3531-istio-token\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:33:58.662091 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.662051 2578 generic.go:356] "Generic (PLEG): container finished" podID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerID="51698a3feae626680ac305084c3105a865300bdb5b684948f3a72edddf61e06b" exitCode=0 Sep 11 14:33:58.662091 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.662092 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" Sep 11 14:33:58.662300 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.662142 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" event={"ID":"445d98e5-207b-43d2-b3e6-601ac81a3531","Type":"ContainerDied","Data":"51698a3feae626680ac305084c3105a865300bdb5b684948f3a72edddf61e06b"} Sep 11 14:33:58.662300 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.662196 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn" event={"ID":"445d98e5-207b-43d2-b3e6-601ac81a3531","Type":"ContainerDied","Data":"dfd9812eb3f2bcbcd69821a3a0cce24ee18ab7ad85c7929e095d02b9c30e1f38"} Sep 11 14:33:58.662300 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.662221 2578 scope.go:117] "RemoveContainer" containerID="51698a3feae626680ac305084c3105a865300bdb5b684948f3a72edddf61e06b" Sep 11 14:33:58.671787 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.671771 2578 scope.go:117] "RemoveContainer" containerID="652444a18d9c854c87f89e485ebfd5981a9ac91feaccbf08fefd482c751dfc08" Sep 11 14:33:58.680452 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.680432 2578 scope.go:117] "RemoveContainer" containerID="51698a3feae626680ac305084c3105a865300bdb5b684948f3a72edddf61e06b" Sep 11 14:33:58.680749 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:33:58.680706 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51698a3feae626680ac305084c3105a865300bdb5b684948f3a72edddf61e06b\": container with ID starting with 51698a3feae626680ac305084c3105a865300bdb5b684948f3a72edddf61e06b not found: ID does not exist" containerID="51698a3feae626680ac305084c3105a865300bdb5b684948f3a72edddf61e06b" Sep 11 14:33:58.680816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.680752 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51698a3feae626680ac305084c3105a865300bdb5b684948f3a72edddf61e06b"} err="failed to get container status \"51698a3feae626680ac305084c3105a865300bdb5b684948f3a72edddf61e06b\": rpc error: code = NotFound desc = could not find container \"51698a3feae626680ac305084c3105a865300bdb5b684948f3a72edddf61e06b\": container with ID starting with 51698a3feae626680ac305084c3105a865300bdb5b684948f3a72edddf61e06b not found: ID does not exist" Sep 11 14:33:58.680816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.680770 2578 scope.go:117] "RemoveContainer" containerID="652444a18d9c854c87f89e485ebfd5981a9ac91feaccbf08fefd482c751dfc08" Sep 11 14:33:58.681017 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:33:58.680996 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652444a18d9c854c87f89e485ebfd5981a9ac91feaccbf08fefd482c751dfc08\": container with ID starting with 652444a18d9c854c87f89e485ebfd5981a9ac91feaccbf08fefd482c751dfc08 not found: ID does not exist" containerID="652444a18d9c854c87f89e485ebfd5981a9ac91feaccbf08fefd482c751dfc08" Sep 11 14:33:58.681067 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.681023 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652444a18d9c854c87f89e485ebfd5981a9ac91feaccbf08fefd482c751dfc08"} err="failed to get container status \"652444a18d9c854c87f89e485ebfd5981a9ac91feaccbf08fefd482c751dfc08\": rpc error: code = NotFound desc = could not find container \"652444a18d9c854c87f89e485ebfd5981a9ac91feaccbf08fefd482c751dfc08\": container with ID starting with 652444a18d9c854c87f89e485ebfd5981a9ac91feaccbf08fefd482c751dfc08 not found: ID does not exist" Sep 11 14:33:58.685121 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.685099 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn"] Sep 11 14:33:58.695137 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:33:58.695111 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-84d3a8-openshift-default-599c789f6b-xs6kn"] Sep 11 14:34:00.263180 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.263147 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr"] Sep 11 14:34:00.263817 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.263797 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerName="istio-proxy" Sep 11 14:34:00.263817 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.263819 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerName="istio-proxy" Sep 11 14:34:00.263978 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.263952 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerName="istio-proxy" Sep 11 14:34:00.263978 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.263966 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerName="istio-proxy" Sep 11 14:34:00.269124 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.269101 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.272038 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.272014 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-259c79-serving-cert\"" Sep 11 14:34:00.272159 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.272046 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:34:00.272159 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.272106 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 14:34:00.273314 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.273298 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 14:34:00.273403 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.273311 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7cfns\"" Sep 11 14:34:00.281695 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.281669 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr"] Sep 11 14:34:00.367714 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.367648 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s2nn\" (UniqueName: \"kubernetes.io/projected/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-kube-api-access-2s2nn\") pod \"maas-api-e2e-explicit-259c79-569c869575-d6vwr\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.367917 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.367823 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-metrics-certs\") pod \"maas-api-e2e-explicit-259c79-569c869575-d6vwr\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.367917 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.367871 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-openshift-service-ca\") pod \"maas-api-e2e-explicit-259c79-569c869575-d6vwr\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.368005 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.367970 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-maas-api-tls\") pod \"maas-api-e2e-explicit-259c79-569c869575-d6vwr\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.465391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.465350 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="445d98e5-207b-43d2-b3e6-601ac81a3531" path="/var/lib/kubelet/pods/445d98e5-207b-43d2-b3e6-601ac81a3531/volumes" Sep 11 14:34:00.468513 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.468490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2nn\" (UniqueName: \"kubernetes.io/projected/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-kube-api-access-2s2nn\") pod \"maas-api-e2e-explicit-259c79-569c869575-d6vwr\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.468621 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.468534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-metrics-certs\") pod \"maas-api-e2e-explicit-259c79-569c869575-d6vwr\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.468621 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.468558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-openshift-service-ca\") pod \"maas-api-e2e-explicit-259c79-569c869575-d6vwr\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.468804 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.468783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-maas-api-tls\") pod \"maas-api-e2e-explicit-259c79-569c869575-d6vwr\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.469218 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.469199 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-openshift-service-ca\") pod \"maas-api-e2e-explicit-259c79-569c869575-d6vwr\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.470965 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.470943 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-metrics-certs\") pod \"maas-api-e2e-explicit-259c79-569c869575-d6vwr\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.471191 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.471163 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-maas-api-tls\") pod \"maas-api-e2e-explicit-259c79-569c869575-d6vwr\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.478842 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.478816 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2nn\" (UniqueName: \"kubernetes.io/projected/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-kube-api-access-2s2nn\") pod \"maas-api-e2e-explicit-259c79-569c869575-d6vwr\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.580941 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.580857 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:00.871519 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:00.871486 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr"] Sep 11 14:34:00.872805 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:34:00.872770 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ec78b3_3fa4_41cd_b175_f2ef3b41d593.slice/crio-610b66a9d3efa8cfff61227de9d1cd09fc5a2024f59d017a571a0519b78c9afe WatchSource:0}: Error finding container 610b66a9d3efa8cfff61227de9d1cd09fc5a2024f59d017a571a0519b78c9afe: Status 404 returned error can't find the container with id 610b66a9d3efa8cfff61227de9d1cd09fc5a2024f59d017a571a0519b78c9afe Sep 11 14:34:01.459189 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.459160 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t"] Sep 11 14:34:01.459587 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.459574 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerName="istio-proxy" Sep 11 14:34:01.459587 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.459589 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="445d98e5-207b-43d2-b3e6-601ac81a3531" containerName="istio-proxy" Sep 11 14:34:01.462821 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.462802 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" Sep 11 14:34:01.465774 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.465755 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-259c79-dockercfg-xzftc\"" Sep 11 14:34:01.465883 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.465829 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-259c79\"" Sep 11 14:34:01.482378 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.482352 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t"] Sep 11 14:34:01.581121 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.581030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrg8v\" (UniqueName: \"kubernetes.io/projected/cd0dacd5-9619-4485-b975-597a2ffd2fbe-kube-api-access-lrg8v\") pod \"payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t\" (UID: \"cd0dacd5-9619-4485-b975-597a2ffd2fbe\") " pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" Sep 11 14:34:01.581121 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.581066 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0dacd5-9619-4485-b975-597a2ffd2fbe-plugins-config-volume\") pod \"payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t\" (UID: \"cd0dacd5-9619-4485-b975-597a2ffd2fbe\") " pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" Sep 11 14:34:01.681591 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.681558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrg8v\" (UniqueName: \"kubernetes.io/projected/cd0dacd5-9619-4485-b975-597a2ffd2fbe-kube-api-access-lrg8v\") pod \"payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t\" (UID: \"cd0dacd5-9619-4485-b975-597a2ffd2fbe\") " pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" Sep 11 14:34:01.681798 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.681607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0dacd5-9619-4485-b975-597a2ffd2fbe-plugins-config-volume\") pod \"payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t\" (UID: \"cd0dacd5-9619-4485-b975-597a2ffd2fbe\") " pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" Sep 11 14:34:01.682521 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.682490 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0dacd5-9619-4485-b975-597a2ffd2fbe-plugins-config-volume\") pod \"payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t\" (UID: \"cd0dacd5-9619-4485-b975-597a2ffd2fbe\") " pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" Sep 11 14:34:01.685216 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.685190 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" event={"ID":"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593","Type":"ContainerStarted","Data":"25d37c89fc950ed7ee11258ad885136d77802beb2cfcf59a5183849c02095975"} Sep 11 14:34:01.685308 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.685243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" event={"ID":"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593","Type":"ContainerStarted","Data":"610b66a9d3efa8cfff61227de9d1cd09fc5a2024f59d017a571a0519b78c9afe"} Sep 11 14:34:01.685355 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.685329 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:01.695642 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.695613 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrg8v\" (UniqueName: \"kubernetes.io/projected/cd0dacd5-9619-4485-b975-597a2ffd2fbe-kube-api-access-lrg8v\") pod \"payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t\" (UID: \"cd0dacd5-9619-4485-b975-597a2ffd2fbe\") " pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" Sep 11 14:34:01.747260 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.747199 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" podStartSLOduration=1.358971621 podStartE2EDuration="1.747177196s" totalImagesPullingTime="388.205575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:34:01.744404999 +0000 UTC m=+1129.829861374" watchObservedRunningTime="2026-09-11 14:34:01.747177196 +0000 UTC m=+1129.832633557" Sep 11 14:34:01.773051 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:01.773020 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" Sep 11 14:34:02.073815 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:02.073781 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t"] Sep 11 14:34:02.074794 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:34:02.074766 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0dacd5_9619_4485_b975_597a2ffd2fbe.slice/crio-894f14bc3fdf8fbcef7096f3c8616074f5544669dce016ed84d86387cd3df5be WatchSource:0}: Error finding container 894f14bc3fdf8fbcef7096f3c8616074f5544669dce016ed84d86387cd3df5be: Status 404 returned error can't find the container with id 894f14bc3fdf8fbcef7096f3c8616074f5544669dce016ed84d86387cd3df5be Sep 11 14:34:02.693667 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:02.693619 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" event={"ID":"cd0dacd5-9619-4485-b975-597a2ffd2fbe","Type":"ContainerStarted","Data":"278dc473dd123d148f157ae074a05a696fbe51a60e1d8e0c4da8522775eb63ce"} Sep 11 14:34:02.694077 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:02.693699 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" event={"ID":"cd0dacd5-9619-4485-b975-597a2ffd2fbe","Type":"ContainerStarted","Data":"894f14bc3fdf8fbcef7096f3c8616074f5544669dce016ed84d86387cd3df5be"} Sep 11 14:34:02.736394 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:02.736347 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" podStartSLOduration=1.7363318840000002 podStartE2EDuration="1.736331884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:34:02.733169644 +0000 UTC m=+1130.818626004" watchObservedRunningTime="2026-09-11 14:34:02.736331884 +0000 UTC m=+1130.821788244" Sep 11 14:34:03.699448 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:03.699418 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" Sep 11 14:34:05.444601 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:05.444570 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:34:07.698645 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:07.698614 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:34:14.707487 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:14.707452 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" Sep 11 14:34:16.788452 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:16.788415 2578 generic.go:356] "Generic (PLEG): container finished" podID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerID="9d62ba5905706a4fec07287d9b5be6b2b05d075decd0b0eb2fa3f5d3b1956e64" exitCode=0 Sep 11 14:34:16.788893 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:16.788485 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" event={"ID":"a02a1fa4-087c-49a8-a89f-f0017b3528f7","Type":"ContainerDied","Data":"9d62ba5905706a4fec07287d9b5be6b2b05d075decd0b0eb2fa3f5d3b1956e64"} Sep 11 14:34:16.788968 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:16.788919 2578 scope.go:117] "RemoveContainer" containerID="9d62ba5905706a4fec07287d9b5be6b2b05d075decd0b0eb2fa3f5d3b1956e64" Sep 11 14:34:16.789952 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:16.789920 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:34:16.790046 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:16.789987 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:34:16.790046 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:16.790018 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:34:17.796708 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:17.796676 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" event={"ID":"a02a1fa4-087c-49a8-a89f-f0017b3528f7","Type":"ContainerStarted","Data":"31ed3c7b6b406df57f345c8bae47f9fe187639e1bc59946e1cdbcd3e6f98498e"} Sep 11 14:34:18.118312 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:18.118277 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:34:18.118520 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:18.118500 2578 patch_prober.go:28] interesting pod/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:34:18.118571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:18.118558 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" podUID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 14:34:19.119313 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:19.119277 2578 patch_prober.go:28] interesting pod/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:34:19.119687 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:19.119336 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" podUID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 14:34:20.118905 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:20.118869 2578 patch_prober.go:28] interesting pod/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:34:20.119075 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:20.118926 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" podUID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 14:34:21.118600 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:21.118563 2578 patch_prober.go:28] interesting pod/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:34:21.119104 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:21.118617 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" podUID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 14:34:22.119273 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:22.119231 2578 patch_prober.go:28] interesting pod/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:34:22.119657 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:22.119291 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" podUID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 14:34:23.122547 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:23.122515 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:34:23.122989 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:23.122926 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:34:23.123504 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:23.123488 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:34:23.527414 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:23.527324 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:34:40.035960 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:40.035883 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:34:54.739370 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:34:54.739333 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:35:10.662675 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:10.662644 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:35:12.366816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:12.366788 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:35:12.367328 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:12.367209 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:35:13.447110 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.447073 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr"] Sep 11 14:35:13.447486 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.447313 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" podUID="b0ec78b3-3fa4-41cd-b175-f2ef3b41d593" containerName="maas-api" containerID="cri-o://25d37c89fc950ed7ee11258ad885136d77802beb2cfcf59a5183849c02095975" gracePeriod=30 Sep 11 14:35:13.519812 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.519780 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t"] Sep 11 14:35:13.520017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.519991 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" podUID="cd0dacd5-9619-4485-b975-597a2ffd2fbe" containerName="payload-processing" containerID="cri-o://278dc473dd123d148f157ae074a05a696fbe51a60e1d8e0c4da8522775eb63ce" gracePeriod=30 Sep 11 14:35:13.772538 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.772512 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:35:13.806262 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.806235 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" Sep 11 14:35:13.945159 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.945128 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0dacd5-9619-4485-b975-597a2ffd2fbe-plugins-config-volume\") pod \"cd0dacd5-9619-4485-b975-597a2ffd2fbe\" (UID: \"cd0dacd5-9619-4485-b975-597a2ffd2fbe\") " Sep 11 14:35:13.945338 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.945168 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s2nn\" (UniqueName: \"kubernetes.io/projected/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-kube-api-access-2s2nn\") pod \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " Sep 11 14:35:13.945338 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.945203 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-openshift-service-ca\") pod \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " Sep 11 14:35:13.945338 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.945275 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-maas-api-tls\") pod \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " Sep 11 14:35:13.945338 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.945324 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrg8v\" (UniqueName: \"kubernetes.io/projected/cd0dacd5-9619-4485-b975-597a2ffd2fbe-kube-api-access-lrg8v\") pod \"cd0dacd5-9619-4485-b975-597a2ffd2fbe\" (UID: \"cd0dacd5-9619-4485-b975-597a2ffd2fbe\") " Sep 11 14:35:13.945554 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.945401 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-metrics-certs\") pod \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\" (UID: \"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593\") " Sep 11 14:35:13.945641 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.945596 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0ec78b3-3fa4-41cd-b175-f2ef3b41d593" (UID: "b0ec78b3-3fa4-41cd-b175-f2ef3b41d593"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:13.945747 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.945696 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0dacd5-9619-4485-b975-597a2ffd2fbe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cd0dacd5-9619-4485-b975-597a2ffd2fbe" (UID: "cd0dacd5-9619-4485-b975-597a2ffd2fbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:13.947465 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.947443 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0ec78b3-3fa4-41cd-b175-f2ef3b41d593" (UID: "b0ec78b3-3fa4-41cd-b175-f2ef3b41d593"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:35:13.947797 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.947773 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0dacd5-9619-4485-b975-597a2ffd2fbe-kube-api-access-lrg8v" (OuterVolumeSpecName: "kube-api-access-lrg8v") pod "cd0dacd5-9619-4485-b975-597a2ffd2fbe" (UID: "cd0dacd5-9619-4485-b975-597a2ffd2fbe"). InnerVolumeSpecName "kube-api-access-lrg8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:13.947887 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.947867 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b0ec78b3-3fa4-41cd-b175-f2ef3b41d593" (UID: "b0ec78b3-3fa4-41cd-b175-f2ef3b41d593"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:35:13.947932 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:13.947905 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-kube-api-access-2s2nn" (OuterVolumeSpecName: "kube-api-access-2s2nn") pod "b0ec78b3-3fa4-41cd-b175-f2ef3b41d593" (UID: "b0ec78b3-3fa4-41cd-b175-f2ef3b41d593"). InnerVolumeSpecName "kube-api-access-2s2nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:14.046509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.046434 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrg8v\" (UniqueName: \"kubernetes.io/projected/cd0dacd5-9619-4485-b975-597a2ffd2fbe-kube-api-access-lrg8v\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:14.046509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.046462 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:14.046509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.046474 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0dacd5-9619-4485-b975-597a2ffd2fbe-plugins-config-volume\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:14.046509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.046482 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s2nn\" (UniqueName: \"kubernetes.io/projected/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-kube-api-access-2s2nn\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:14.046509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.046491 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:14.046509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.046499 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:14.136309 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.136273 2578 generic.go:356] "Generic (PLEG): container finished" podID="b0ec78b3-3fa4-41cd-b175-f2ef3b41d593" containerID="25d37c89fc950ed7ee11258ad885136d77802beb2cfcf59a5183849c02095975" exitCode=0 Sep 11 14:35:14.136492 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.136322 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" Sep 11 14:35:14.136492 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.136365 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" event={"ID":"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593","Type":"ContainerDied","Data":"25d37c89fc950ed7ee11258ad885136d77802beb2cfcf59a5183849c02095975"} Sep 11 14:35:14.136492 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.136422 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr" event={"ID":"b0ec78b3-3fa4-41cd-b175-f2ef3b41d593","Type":"ContainerDied","Data":"610b66a9d3efa8cfff61227de9d1cd09fc5a2024f59d017a571a0519b78c9afe"} Sep 11 14:35:14.136492 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.136450 2578 scope.go:117] "RemoveContainer" containerID="25d37c89fc950ed7ee11258ad885136d77802beb2cfcf59a5183849c02095975" Sep 11 14:35:14.137921 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.137896 2578 generic.go:356] "Generic (PLEG): container finished" podID="cd0dacd5-9619-4485-b975-597a2ffd2fbe" containerID="278dc473dd123d148f157ae074a05a696fbe51a60e1d8e0c4da8522775eb63ce" exitCode=0 Sep 11 14:35:14.138051 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.137948 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" Sep 11 14:35:14.138051 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.137990 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" event={"ID":"cd0dacd5-9619-4485-b975-597a2ffd2fbe","Type":"ContainerDied","Data":"278dc473dd123d148f157ae074a05a696fbe51a60e1d8e0c4da8522775eb63ce"} Sep 11 14:35:14.138162 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.138060 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t" event={"ID":"cd0dacd5-9619-4485-b975-597a2ffd2fbe","Type":"ContainerDied","Data":"894f14bc3fdf8fbcef7096f3c8616074f5544669dce016ed84d86387cd3df5be"} Sep 11 14:35:14.147716 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.147697 2578 scope.go:117] "RemoveContainer" containerID="25d37c89fc950ed7ee11258ad885136d77802beb2cfcf59a5183849c02095975" Sep 11 14:35:14.148097 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:35:14.148077 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d37c89fc950ed7ee11258ad885136d77802beb2cfcf59a5183849c02095975\": container with ID starting with 25d37c89fc950ed7ee11258ad885136d77802beb2cfcf59a5183849c02095975 not found: ID does not exist" containerID="25d37c89fc950ed7ee11258ad885136d77802beb2cfcf59a5183849c02095975" Sep 11 14:35:14.148150 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.148106 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d37c89fc950ed7ee11258ad885136d77802beb2cfcf59a5183849c02095975"} err="failed to get container status \"25d37c89fc950ed7ee11258ad885136d77802beb2cfcf59a5183849c02095975\": rpc error: code = NotFound desc = could not find container \"25d37c89fc950ed7ee11258ad885136d77802beb2cfcf59a5183849c02095975\": container with ID starting with 25d37c89fc950ed7ee11258ad885136d77802beb2cfcf59a5183849c02095975 not found: ID does not exist" Sep 11 14:35:14.148150 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.148123 2578 scope.go:117] "RemoveContainer" containerID="278dc473dd123d148f157ae074a05a696fbe51a60e1d8e0c4da8522775eb63ce" Sep 11 14:35:14.157525 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.157507 2578 scope.go:117] "RemoveContainer" containerID="278dc473dd123d148f157ae074a05a696fbe51a60e1d8e0c4da8522775eb63ce" Sep 11 14:35:14.157845 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:35:14.157822 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"278dc473dd123d148f157ae074a05a696fbe51a60e1d8e0c4da8522775eb63ce\": container with ID starting with 278dc473dd123d148f157ae074a05a696fbe51a60e1d8e0c4da8522775eb63ce not found: ID does not exist" containerID="278dc473dd123d148f157ae074a05a696fbe51a60e1d8e0c4da8522775eb63ce" Sep 11 14:35:14.157930 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.157852 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278dc473dd123d148f157ae074a05a696fbe51a60e1d8e0c4da8522775eb63ce"} err="failed to get container status \"278dc473dd123d148f157ae074a05a696fbe51a60e1d8e0c4da8522775eb63ce\": rpc error: code = NotFound desc = could not find container \"278dc473dd123d148f157ae074a05a696fbe51a60e1d8e0c4da8522775eb63ce\": container with ID starting with 278dc473dd123d148f157ae074a05a696fbe51a60e1d8e0c4da8522775eb63ce not found: ID does not exist" Sep 11 14:35:14.178061 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.178037 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr"] Sep 11 14:35:14.189895 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.189871 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-259c79-569c869575-d6vwr"] Sep 11 14:35:14.226017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.225987 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t"] Sep 11 14:35:14.229708 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.229679 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-259c79-85b4b8754f-9rt5t"] Sep 11 14:35:14.464804 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.464777 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ec78b3-3fa4-41cd-b175-f2ef3b41d593" path="/var/lib/kubelet/pods/b0ec78b3-3fa4-41cd-b175-f2ef3b41d593/volumes" Sep 11 14:35:14.465307 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:14.465286 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd0dacd5-9619-4485-b975-597a2ffd2fbe" path="/var/lib/kubelet/pods/cd0dacd5-9619-4485-b975-597a2ffd2fbe/volumes" Sep 11 14:35:21.569676 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:21.569641 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:35:24.337334 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:24.337297 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx"] Sep 11 14:35:24.337837 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:24.337518 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" podUID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerName="istio-proxy" containerID="cri-o://31ed3c7b6b406df57f345c8bae47f9fe187639e1bc59946e1cdbcd3e6f98498e" gracePeriod=30 Sep 11 14:35:29.585468 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.585443 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:35:29.680560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.680475 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfns2\" (UniqueName: \"kubernetes.io/projected/a02a1fa4-087c-49a8-a89f-f0017b3528f7-kube-api-access-xfns2\") pod \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " Sep 11 14:35:29.680560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.680536 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-workload-certs\") pod \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " Sep 11 14:35:29.680560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.680554 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-podinfo\") pod \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " Sep 11 14:35:29.680893 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.680575 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-token\") pod \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " Sep 11 14:35:29.680893 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.680594 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-data\") pod \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " Sep 11 14:35:29.680893 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.680627 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-credential-socket\") pod \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " Sep 11 14:35:29.680893 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.680657 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-envoy\") pod \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " Sep 11 14:35:29.680893 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.680712 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-workload-socket\") pod \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " Sep 11 14:35:29.680893 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.680750 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istiod-ca-cert\") pod \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\" (UID: \"a02a1fa4-087c-49a8-a89f-f0017b3528f7\") " Sep 11 14:35:29.681213 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.681000 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a02a1fa4-087c-49a8-a89f-f0017b3528f7" (UID: "a02a1fa4-087c-49a8-a89f-f0017b3528f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:35:29.681213 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.681118 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a02a1fa4-087c-49a8-a89f-f0017b3528f7" (UID: "a02a1fa4-087c-49a8-a89f-f0017b3528f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:35:29.681323 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.681244 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a02a1fa4-087c-49a8-a89f-f0017b3528f7" (UID: "a02a1fa4-087c-49a8-a89f-f0017b3528f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:35:29.681323 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.681313 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a02a1fa4-087c-49a8-a89f-f0017b3528f7" (UID: "a02a1fa4-087c-49a8-a89f-f0017b3528f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:29.681455 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.681428 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "a02a1fa4-087c-49a8-a89f-f0017b3528f7" (UID: "a02a1fa4-087c-49a8-a89f-f0017b3528f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:35:29.682937 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.682911 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "a02a1fa4-087c-49a8-a89f-f0017b3528f7" (UID: "a02a1fa4-087c-49a8-a89f-f0017b3528f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:29.683251 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.683223 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a02a1fa4-087c-49a8-a89f-f0017b3528f7" (UID: "a02a1fa4-087c-49a8-a89f-f0017b3528f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:35:29.683251 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.683236 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02a1fa4-087c-49a8-a89f-f0017b3528f7-kube-api-access-xfns2" (OuterVolumeSpecName: "kube-api-access-xfns2") pod "a02a1fa4-087c-49a8-a89f-f0017b3528f7" (UID: "a02a1fa4-087c-49a8-a89f-f0017b3528f7"). InnerVolumeSpecName "kube-api-access-xfns2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:29.683390 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.683244 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a02a1fa4-087c-49a8-a89f-f0017b3528f7" (UID: "a02a1fa4-087c-49a8-a89f-f0017b3528f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:35:29.697423 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.697398 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:35:29.781576 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.781536 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-token\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:29.781576 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.781569 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-data\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:29.781576 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.781580 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-credential-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:29.781576 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.781589 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-envoy\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:29.781941 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.781599 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-workload-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:29.781941 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.781608 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istiod-ca-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:29.781941 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.781617 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfns2\" (UniqueName: \"kubernetes.io/projected/a02a1fa4-087c-49a8-a89f-f0017b3528f7-kube-api-access-xfns2\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:29.781941 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.781625 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a02a1fa4-087c-49a8-a89f-f0017b3528f7-workload-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:29.781941 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:29.781634 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a02a1fa4-087c-49a8-a89f-f0017b3528f7-istio-podinfo\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:30.239031 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:30.238996 2578 generic.go:356] "Generic (PLEG): container finished" podID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerID="31ed3c7b6b406df57f345c8bae47f9fe187639e1bc59946e1cdbcd3e6f98498e" exitCode=0 Sep 11 14:35:30.239228 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:30.239074 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" Sep 11 14:35:30.239228 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:30.239078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" event={"ID":"a02a1fa4-087c-49a8-a89f-f0017b3528f7","Type":"ContainerDied","Data":"31ed3c7b6b406df57f345c8bae47f9fe187639e1bc59946e1cdbcd3e6f98498e"} Sep 11 14:35:30.239228 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:30.239132 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx" event={"ID":"a02a1fa4-087c-49a8-a89f-f0017b3528f7","Type":"ContainerDied","Data":"1d5e7fcee9def3ec330399a76dce89f79cce6515d629b8aeddc13d268b4bf769"} Sep 11 14:35:30.239228 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:30.239154 2578 scope.go:117] "RemoveContainer" containerID="31ed3c7b6b406df57f345c8bae47f9fe187639e1bc59946e1cdbcd3e6f98498e" Sep 11 14:35:30.249199 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:30.249177 2578 scope.go:117] "RemoveContainer" containerID="9d62ba5905706a4fec07287d9b5be6b2b05d075decd0b0eb2fa3f5d3b1956e64" Sep 11 14:35:30.258331 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:30.258314 2578 scope.go:117] "RemoveContainer" containerID="31ed3c7b6b406df57f345c8bae47f9fe187639e1bc59946e1cdbcd3e6f98498e" Sep 11 14:35:30.258583 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:35:30.258564 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ed3c7b6b406df57f345c8bae47f9fe187639e1bc59946e1cdbcd3e6f98498e\": container with ID starting with 31ed3c7b6b406df57f345c8bae47f9fe187639e1bc59946e1cdbcd3e6f98498e not found: ID does not exist" containerID="31ed3c7b6b406df57f345c8bae47f9fe187639e1bc59946e1cdbcd3e6f98498e" Sep 11 14:35:30.258647 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:30.258589 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ed3c7b6b406df57f345c8bae47f9fe187639e1bc59946e1cdbcd3e6f98498e"} err="failed to get container status \"31ed3c7b6b406df57f345c8bae47f9fe187639e1bc59946e1cdbcd3e6f98498e\": rpc error: code = NotFound desc = could not find container \"31ed3c7b6b406df57f345c8bae47f9fe187639e1bc59946e1cdbcd3e6f98498e\": container with ID starting with 31ed3c7b6b406df57f345c8bae47f9fe187639e1bc59946e1cdbcd3e6f98498e not found: ID does not exist" Sep 11 14:35:30.258647 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:30.258605 2578 scope.go:117] "RemoveContainer" containerID="9d62ba5905706a4fec07287d9b5be6b2b05d075decd0b0eb2fa3f5d3b1956e64" Sep 11 14:35:30.258876 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:35:30.258860 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d62ba5905706a4fec07287d9b5be6b2b05d075decd0b0eb2fa3f5d3b1956e64\": container with ID starting with 9d62ba5905706a4fec07287d9b5be6b2b05d075decd0b0eb2fa3f5d3b1956e64 not found: ID does not exist" containerID="9d62ba5905706a4fec07287d9b5be6b2b05d075decd0b0eb2fa3f5d3b1956e64" Sep 11 14:35:30.258923 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:30.258881 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d62ba5905706a4fec07287d9b5be6b2b05d075decd0b0eb2fa3f5d3b1956e64"} err="failed to get container status \"9d62ba5905706a4fec07287d9b5be6b2b05d075decd0b0eb2fa3f5d3b1956e64\": rpc error: code = NotFound desc = could not find container \"9d62ba5905706a4fec07287d9b5be6b2b05d075decd0b0eb2fa3f5d3b1956e64\": container with ID starting with 9d62ba5905706a4fec07287d9b5be6b2b05d075decd0b0eb2fa3f5d3b1956e64 not found: ID does not exist" Sep 11 14:35:30.282119 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:30.282092 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx"] Sep 11 14:35:30.291359 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:30.291338 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-259c79-openshift-default-f54f7b4c6-nx6sx"] Sep 11 14:35:30.464186 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:30.464156 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" path="/var/lib/kubelet/pods/a02a1fa4-087c-49a8-a89f-f0017b3528f7/volumes" Sep 11 14:35:31.328553 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.328522 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw"] Sep 11 14:35:31.328994 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.328940 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerName="istio-proxy" Sep 11 14:35:31.328994 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.328953 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerName="istio-proxy" Sep 11 14:35:31.328994 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.328970 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ec78b3-3fa4-41cd-b175-f2ef3b41d593" containerName="maas-api" Sep 11 14:35:31.328994 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.328975 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ec78b3-3fa4-41cd-b175-f2ef3b41d593" containerName="maas-api" Sep 11 14:35:31.328994 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.328986 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd0dacd5-9619-4485-b975-597a2ffd2fbe" containerName="payload-processing" Sep 11 14:35:31.328994 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.328992 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd0dacd5-9619-4485-b975-597a2ffd2fbe" containerName="payload-processing" Sep 11 14:35:31.329212 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.329052 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd0dacd5-9619-4485-b975-597a2ffd2fbe" containerName="payload-processing" Sep 11 14:35:31.329212 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.329061 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ec78b3-3fa4-41cd-b175-f2ef3b41d593" containerName="maas-api" Sep 11 14:35:31.329212 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.329070 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerName="istio-proxy" Sep 11 14:35:31.329212 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.329078 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerName="istio-proxy" Sep 11 14:35:31.333741 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.333705 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw" Sep 11 14:35:31.337247 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.337224 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:35:31.337378 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.337224 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\"" Sep 11 14:35:31.353037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.353006 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw"] Sep 11 14:35:31.395095 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.395068 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw\" (UID: \"bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw" Sep 11 14:35:31.496252 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.496224 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw\" (UID: \"bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw" Sep 11 14:35:31.496790 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.496773 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw\" (UID: \"bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw" Sep 11 14:35:31.643801 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.643773 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw" Sep 11 14:35:31.943027 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:31.943002 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw"] Sep 11 14:35:31.945052 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:35:31.945024 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf72a11_46fe_4f68_bbdd_cbf5a35e14b1.slice/crio-9a2d6b0c961684e3b9392f0eb6aa3dfe47f62793dae9567ae92c2e8e1c269357 WatchSource:0}: Error finding container 9a2d6b0c961684e3b9392f0eb6aa3dfe47f62793dae9567ae92c2e8e1c269357: Status 404 returned error can't find the container with id 9a2d6b0c961684e3b9392f0eb6aa3dfe47f62793dae9567ae92c2e8e1c269357 Sep 11 14:35:32.255439 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:32.255365 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw" event={"ID":"bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1","Type":"ContainerStarted","Data":"79741c22aa1c8fcc80016baa10fc7bd7367847e03fa010871deb72f28e31ec1c"} Sep 11 14:35:32.255439 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:32.255415 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw" event={"ID":"bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1","Type":"ContainerStarted","Data":"9a2d6b0c961684e3b9392f0eb6aa3dfe47f62793dae9567ae92c2e8e1c269357"} Sep 11 14:35:32.286332 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:32.286284 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw" podStartSLOduration=1.286270907 podStartE2EDuration="1.286270907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:35:32.285029266 +0000 UTC m=+1220.370485630" watchObservedRunningTime="2026-09-11 14:35:32.286270907 +0000 UTC m=+1220.371727267" Sep 11 14:35:33.263308 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:33.263274 2578 generic.go:356] "Generic (PLEG): container finished" podID="bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1" containerID="79741c22aa1c8fcc80016baa10fc7bd7367847e03fa010871deb72f28e31ec1c" exitCode=0 Sep 11 14:35:33.263704 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:33.263312 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw" event={"ID":"bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1","Type":"ContainerDied","Data":"79741c22aa1c8fcc80016baa10fc7bd7367847e03fa010871deb72f28e31ec1c"} Sep 11 14:35:34.408983 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:34.408959 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw" Sep 11 14:35:34.523799 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:34.523696 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1-maas-api-service-ca\") pod \"bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1\" (UID: \"bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1\") " Sep 11 14:35:34.524014 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:34.523992 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1" (UID: "bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:34.625098 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:34.625072 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1-maas-api-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:35.276314 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:35.276281 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw" event={"ID":"bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1","Type":"ContainerDied","Data":"9a2d6b0c961684e3b9392f0eb6aa3dfe47f62793dae9567ae92c2e8e1c269357"} Sep 11 14:35:35.276485 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:35.276330 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a2d6b0c961684e3b9392f0eb6aa3dfe47f62793dae9567ae92c2e8e1c269357" Sep 11 14:35:35.276485 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:35.276288 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw" Sep 11 14:35:41.030512 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:41.030477 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:35:42.265330 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:42.265296 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd"] Sep 11 14:35:42.265819 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:42.265537 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" podUID="cbca1efd-7b92-4bba-8937-2aa60d7dc4e8" containerName="payload-processing" containerID="cri-o://580799353aa1f3ccef77d480c37bc432230a5e8567afe4bc7c97e766d73ef6c1" gracePeriod=30 Sep 11 14:35:42.526275 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:42.526216 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" Sep 11 14:35:42.595794 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:42.595763 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlshm\" (UniqueName: \"kubernetes.io/projected/cbca1efd-7b92-4bba-8937-2aa60d7dc4e8-kube-api-access-dlshm\") pod \"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8\" (UID: \"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8\") " Sep 11 14:35:42.595959 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:42.595882 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbca1efd-7b92-4bba-8937-2aa60d7dc4e8-plugins-config-volume\") pod \"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8\" (UID: \"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8\") " Sep 11 14:35:42.596185 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:42.596158 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbca1efd-7b92-4bba-8937-2aa60d7dc4e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cbca1efd-7b92-4bba-8937-2aa60d7dc4e8" (UID: "cbca1efd-7b92-4bba-8937-2aa60d7dc4e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:42.598140 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:42.598119 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbca1efd-7b92-4bba-8937-2aa60d7dc4e8-kube-api-access-dlshm" (OuterVolumeSpecName: "kube-api-access-dlshm") pod "cbca1efd-7b92-4bba-8937-2aa60d7dc4e8" (UID: "cbca1efd-7b92-4bba-8937-2aa60d7dc4e8"). InnerVolumeSpecName "kube-api-access-dlshm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:42.599587 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:42.599568 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw"] Sep 11 14:35:42.609696 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:42.609674 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7035-djcgw"] Sep 11 14:35:42.697232 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:42.697204 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbca1efd-7b92-4bba-8937-2aa60d7dc4e8-plugins-config-volume\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:42.697232 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:42.697227 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlshm\" (UniqueName: \"kubernetes.io/projected/cbca1efd-7b92-4bba-8937-2aa60d7dc4e8-kube-api-access-dlshm\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:35:43.324322 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:43.324286 2578 generic.go:356] "Generic (PLEG): container finished" podID="cbca1efd-7b92-4bba-8937-2aa60d7dc4e8" containerID="580799353aa1f3ccef77d480c37bc432230a5e8567afe4bc7c97e766d73ef6c1" exitCode=0 Sep 11 14:35:43.324322 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:43.324323 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" Sep 11 14:35:43.324862 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:43.324385 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" event={"ID":"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8","Type":"ContainerDied","Data":"580799353aa1f3ccef77d480c37bc432230a5e8567afe4bc7c97e766d73ef6c1"} Sep 11 14:35:43.324862 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:43.324438 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd" event={"ID":"cbca1efd-7b92-4bba-8937-2aa60d7dc4e8","Type":"ContainerDied","Data":"3e57612a13b0f64ffbdc007458350e3428c54e860e3261adc8c26ff7bed956a0"} Sep 11 14:35:43.324862 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:43.324462 2578 scope.go:117] "RemoveContainer" containerID="580799353aa1f3ccef77d480c37bc432230a5e8567afe4bc7c97e766d73ef6c1" Sep 11 14:35:43.334240 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:43.334220 2578 scope.go:117] "RemoveContainer" containerID="580799353aa1f3ccef77d480c37bc432230a5e8567afe4bc7c97e766d73ef6c1" Sep 11 14:35:43.334519 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:35:43.334500 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580799353aa1f3ccef77d480c37bc432230a5e8567afe4bc7c97e766d73ef6c1\": container with ID starting with 580799353aa1f3ccef77d480c37bc432230a5e8567afe4bc7c97e766d73ef6c1 not found: ID does not exist" containerID="580799353aa1f3ccef77d480c37bc432230a5e8567afe4bc7c97e766d73ef6c1" Sep 11 14:35:43.334593 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:43.334526 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580799353aa1f3ccef77d480c37bc432230a5e8567afe4bc7c97e766d73ef6c1"} err="failed to get container status \"580799353aa1f3ccef77d480c37bc432230a5e8567afe4bc7c97e766d73ef6c1\": rpc error: code = NotFound desc = could not find container \"580799353aa1f3ccef77d480c37bc432230a5e8567afe4bc7c97e766d73ef6c1\": container with ID starting with 580799353aa1f3ccef77d480c37bc432230a5e8567afe4bc7c97e766d73ef6c1 not found: ID does not exist" Sep 11 14:35:43.361872 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:43.361841 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd"] Sep 11 14:35:43.372086 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:43.372066 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5a7035-5869b9985c-8tmjd"] Sep 11 14:35:44.465302 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:44.465270 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1" path="/var/lib/kubelet/pods/bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1/volumes" Sep 11 14:35:44.465669 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:44.465635 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbca1efd-7b92-4bba-8937-2aa60d7dc4e8" path="/var/lib/kubelet/pods/cbca1efd-7b92-4bba-8937-2aa60d7dc4e8/volumes" Sep 11 14:35:47.396239 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.396202 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg"] Sep 11 14:35:47.396627 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.396614 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerName="istio-proxy" Sep 11 14:35:47.396672 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.396628 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a02a1fa4-087c-49a8-a89f-f0017b3528f7" containerName="istio-proxy" Sep 11 14:35:47.396672 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.396643 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbca1efd-7b92-4bba-8937-2aa60d7dc4e8" containerName="payload-processing" Sep 11 14:35:47.396672 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.396649 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbca1efd-7b92-4bba-8937-2aa60d7dc4e8" containerName="payload-processing" Sep 11 14:35:47.396672 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.396655 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1" containerName="revoke-keys" Sep 11 14:35:47.396672 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.396660 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1" containerName="revoke-keys" Sep 11 14:35:47.396844 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.396753 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdf72a11-46fe-4f68-bbdd-cbf5a35e14b1" containerName="revoke-keys" Sep 11 14:35:47.396844 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.396764 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbca1efd-7b92-4bba-8937-2aa60d7dc4e8" containerName="payload-processing" Sep 11 14:35:47.401322 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.401305 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.404754 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.404716 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c3340176-openshift-default-dockercfg-k5jw6\"" Sep 11 14:35:47.405035 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.405017 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 14:35:47.516204 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.516173 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg"] Sep 11 14:35:47.539781 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.539752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-podinfo\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.539909 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.539789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-envoy\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.539909 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.539861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istiod-ca-cert\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.539909 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.539891 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-credential-socket\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.540017 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.539947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-workload-socket\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.540060 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.540048 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-workload-certs\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.540098 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.540077 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-data\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.540133 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.540100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-token\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.540167 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.540133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6vl7\" (UniqueName: \"kubernetes.io/projected/a7cb9caa-f3a6-499a-8213-09c8d5b12903-kube-api-access-k6vl7\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.641340 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.641316 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istiod-ca-cert\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.641474 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.641346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-credential-socket\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.641474 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.641381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-workload-socket\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.641474 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.641400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-workload-certs\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.641474 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.641423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-data\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.641474 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.641445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-token\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.641769 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.641484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vl7\" (UniqueName: \"kubernetes.io/projected/a7cb9caa-f3a6-499a-8213-09c8d5b12903-kube-api-access-k6vl7\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.641769 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.641525 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-podinfo\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.641769 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.641556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-envoy\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.641883 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.641794 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-workload-socket\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.641953 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.641932 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-data\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.642080 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.642058 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-workload-certs\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.642202 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.642179 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-credential-socket\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.642400 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.642382 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istiod-ca-cert\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.643940 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.643911 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-envoy\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.644069 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.644051 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-podinfo\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.656337 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.656280 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vl7\" (UniqueName: \"kubernetes.io/projected/a7cb9caa-f3a6-499a-8213-09c8d5b12903-kube-api-access-k6vl7\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.657872 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.657849 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-token\") pod \"e2e-mt-c3340176-openshift-default-7c4c495788-4fplg\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:47.710889 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:47.710867 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:48.002340 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:48.002304 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg"] Sep 11 14:35:48.003066 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:35:48.003036 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cb9caa_f3a6_499a_8213_09c8d5b12903.slice/crio-1506f26f40bdd6f1643452c4516e8334b23fdd19e534f12aadae328f02c75802 WatchSource:0}: Error finding container 1506f26f40bdd6f1643452c4516e8334b23fdd19e534f12aadae328f02c75802: Status 404 returned error can't find the container with id 1506f26f40bdd6f1643452c4516e8334b23fdd19e534f12aadae328f02c75802 Sep 11 14:35:48.005266 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:48.005237 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:35:48.005340 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:48.005301 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:35:48.005340 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:48.005328 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:35:48.357610 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:48.357577 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" event={"ID":"a7cb9caa-f3a6-499a-8213-09c8d5b12903","Type":"ContainerStarted","Data":"9956364e34aebab4b2089f0125ec5afca11e34929330a9ff4ca6779db7dd002f"} Sep 11 14:35:48.357825 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:48.357646 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" event={"ID":"a7cb9caa-f3a6-499a-8213-09c8d5b12903","Type":"ContainerStarted","Data":"1506f26f40bdd6f1643452c4516e8334b23fdd19e534f12aadae328f02c75802"} Sep 11 14:35:48.398765 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:48.398705 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" podStartSLOduration=1.398690756 podStartE2EDuration="1.398690756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:35:48.396205455 +0000 UTC m=+1236.481661853" watchObservedRunningTime="2026-09-11 14:35:48.398690756 +0000 UTC m=+1236.484147116" Sep 11 14:35:48.711149 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:48.711057 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:49.716210 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:49.716184 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:49.801891 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:49.801860 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:35:50.370266 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:50.370227 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:50.371199 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:50.371182 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:35:51.646967 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.646932 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq"] Sep 11 14:35:51.650806 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.650782 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:51.654958 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.654933 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 14:35:51.655089 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.655011 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7cfns\"" Sep 11 14:35:51.656324 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.656298 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 14:35:51.656471 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.656454 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c3340176-serving-cert\"" Sep 11 14:35:51.656542 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.656528 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:35:51.668474 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.668453 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq"] Sep 11 14:35:51.776631 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.776603 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd2fcdf5-39c6-450e-975f-850f88e4ba29-openshift-service-ca\") pod \"maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:51.776631 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.776631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x62d5\" (UniqueName: \"kubernetes.io/projected/cd2fcdf5-39c6-450e-975f-850f88e4ba29-kube-api-access-x62d5\") pod \"maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:51.776857 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.776757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd2fcdf5-39c6-450e-975f-850f88e4ba29-metrics-certs\") pod \"maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:51.776857 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.776786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd2fcdf5-39c6-450e-975f-850f88e4ba29-maas-api-tls\") pod \"maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:51.877810 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.877779 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd2fcdf5-39c6-450e-975f-850f88e4ba29-openshift-service-ca\") pod \"maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:51.877810 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.877810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x62d5\" (UniqueName: \"kubernetes.io/projected/cd2fcdf5-39c6-450e-975f-850f88e4ba29-kube-api-access-x62d5\") pod \"maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:51.877994 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.877873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd2fcdf5-39c6-450e-975f-850f88e4ba29-metrics-certs\") pod \"maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:51.877994 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.877894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd2fcdf5-39c6-450e-975f-850f88e4ba29-maas-api-tls\") pod \"maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:51.878481 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.878456 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd2fcdf5-39c6-450e-975f-850f88e4ba29-openshift-service-ca\") pod \"maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:51.880382 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.880355 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd2fcdf5-39c6-450e-975f-850f88e4ba29-maas-api-tls\") pod \"maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:51.880535 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.880517 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd2fcdf5-39c6-450e-975f-850f88e4ba29-metrics-certs\") pod \"maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:51.891975 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.891956 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x62d5\" (UniqueName: \"kubernetes.io/projected/cd2fcdf5-39c6-450e-975f-850f88e4ba29-kube-api-access-x62d5\") pod \"maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:51.964099 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:51.963998 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:52.255752 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:52.255704 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq"] Sep 11 14:35:52.257505 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:35:52.257462 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2fcdf5_39c6_450e_975f_850f88e4ba29.slice/crio-eafbe285346a5cec6355b8d762003af63729a8f801546d7473bb6278f06ab604 WatchSource:0}: Error finding container eafbe285346a5cec6355b8d762003af63729a8f801546d7473bb6278f06ab604: Status 404 returned error can't find the container with id eafbe285346a5cec6355b8d762003af63729a8f801546d7473bb6278f06ab604 Sep 11 14:35:52.383662 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:52.383623 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" event={"ID":"cd2fcdf5-39c6-450e-975f-850f88e4ba29","Type":"ContainerStarted","Data":"eafbe285346a5cec6355b8d762003af63729a8f801546d7473bb6278f06ab604"} Sep 11 14:35:53.392378 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:53.392342 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" event={"ID":"cd2fcdf5-39c6-450e-975f-850f88e4ba29","Type":"ContainerStarted","Data":"a56482cc5eece74156e7a42d4186482ada7748cca56109df6ccd237440b7c88b"} Sep 11 14:35:53.392796 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:53.392481 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:35:53.428333 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:53.428276 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" podStartSLOduration=1.642889159 podStartE2EDuration="2.428261082s" totalImagesPullingTime="785.371923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:35:53.424969237 +0000 UTC m=+1241.510425597" watchObservedRunningTime="2026-09-11 14:35:53.428261082 +0000 UTC m=+1241.513717442" Sep 11 14:35:59.404454 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:35:59.404420 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:36:03.972430 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:03.972391 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:36:05.914194 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:05.913765 2578 patch_prober.go:28] interesting pod/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": read tcp 10.134.0.2:55478->10.134.0.74:15021: read: connection reset by peer" start-of-body= Sep 11 14:36:05.914194 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:05.913832 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": read tcp 10.134.0.2:55478->10.134.0.74:15021: read: connection reset by peer" Sep 11 14:36:06.481411 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:06.481377 2578 generic.go:356] "Generic (PLEG): container finished" podID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerID="9956364e34aebab4b2089f0125ec5afca11e34929330a9ff4ca6779db7dd002f" exitCode=0 Sep 11 14:36:06.481587 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:06.481449 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" event={"ID":"a7cb9caa-f3a6-499a-8213-09c8d5b12903","Type":"ContainerDied","Data":"9956364e34aebab4b2089f0125ec5afca11e34929330a9ff4ca6779db7dd002f"} Sep 11 14:36:06.481830 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:06.481814 2578 scope.go:117] "RemoveContainer" containerID="9956364e34aebab4b2089f0125ec5afca11e34929330a9ff4ca6779db7dd002f" Sep 11 14:36:06.482892 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:06.482862 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:36:06.482975 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:06.482942 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:36:06.483035 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:06.482974 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:36:07.492443 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:07.492405 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" event={"ID":"a7cb9caa-f3a6-499a-8213-09c8d5b12903","Type":"ContainerStarted","Data":"60a8c38201ccc85fc22aecae792278e190a1203e8e9eb9e0fd26b252c3134603"} Sep 11 14:36:07.711677 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:07.711644 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:36:07.712007 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:07.711985 2578 patch_prober.go:28] interesting pod/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 11 14:36:07.712065 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:07.712047 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 11 14:36:08.711684 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:08.711648 2578 patch_prober.go:28] interesting pod/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 11 14:36:08.712087 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:08.711704 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 11 14:36:09.711923 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:09.711889 2578 patch_prober.go:28] interesting pod/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 11 14:36:09.712296 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:09.711953 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 11 14:36:10.712018 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:10.711983 2578 patch_prober.go:28] interesting pod/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 11 14:36:10.712380 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:10.712038 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 11 14:36:11.712043 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:11.712005 2578 patch_prober.go:28] interesting pod/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 11 14:36:11.712441 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:11.712065 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 11 14:36:12.711602 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:12.711570 2578 patch_prober.go:28] interesting pod/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 11 14:36:12.711795 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:12.711622 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 11 14:36:13.715452 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:13.715422 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:36:13.715833 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:13.715698 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:36:13.716441 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:13.716425 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:36:27.967299 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:27.967260 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d"] Sep 11 14:36:27.973177 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:27.973152 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:27.976029 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:27.976000 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-1c284e-serving-cert\"" Sep 11 14:36:27.986192 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:27.986166 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d"] Sep 11 14:36:27.998478 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:27.998454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:27.998599 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:27.998482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:27.998599 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:27.998527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcd8b\" (UniqueName: \"kubernetes.io/projected/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-kube-api-access-kcd8b\") pod \"maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:27.998599 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:27.998564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:28.099700 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:28.099667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:28.099700 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:28.099702 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:28.099956 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:28.099782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcd8b\" (UniqueName: \"kubernetes.io/projected/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-kube-api-access-kcd8b\") pod \"maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:28.099956 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:28.099811 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:28.100552 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:28.100527 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:28.102396 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:28.102370 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:28.102525 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:28.102506 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:28.111126 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:28.111107 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcd8b\" (UniqueName: \"kubernetes.io/projected/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-kube-api-access-kcd8b\") pod \"maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:28.284764 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:28.284665 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:28.564289 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:28.564205 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d"] Sep 11 14:36:28.567505 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:36:28.567474 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd34af71_497b_4c4e_a664_8cf8c4a1e9f2.slice/crio-c962cfda97ebf83c99fe24265c68ee3abc44400b8f3ea520a7d0f7d18fd8078f WatchSource:0}: Error finding container c962cfda97ebf83c99fe24265c68ee3abc44400b8f3ea520a7d0f7d18fd8078f: Status 404 returned error can't find the container with id c962cfda97ebf83c99fe24265c68ee3abc44400b8f3ea520a7d0f7d18fd8078f Sep 11 14:36:28.569368 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:28.569344 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:36:28.614315 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:28.614292 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" event={"ID":"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2","Type":"ContainerStarted","Data":"c962cfda97ebf83c99fe24265c68ee3abc44400b8f3ea520a7d0f7d18fd8078f"} Sep 11 14:36:29.628156 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:29.628117 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" event={"ID":"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2","Type":"ContainerStarted","Data":"1e0a2e322eb7dff328eca5f8b4d49ca2363d68ff9da5e9e18398c6b4cffd4a80"} Sep 11 14:36:29.628529 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:29.628213 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:29.651871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:29.651827 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" podStartSLOduration=2.339965216 podStartE2EDuration="2.651812651s" totalImagesPullingTime="311.847435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:36:29.649675945 +0000 UTC m=+1277.735132306" watchObservedRunningTime="2026-09-11 14:36:29.651812651 +0000 UTC m=+1277.737269011" Sep 11 14:36:32.237347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:32.237311 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:36:35.639171 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:35.639141 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:36:37.167469 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.167424 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq"] Sep 11 14:36:37.167957 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.167724 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" podUID="cd2fcdf5-39c6-450e-975f-850f88e4ba29" containerName="maas-api" containerID="cri-o://a56482cc5eece74156e7a42d4186482ada7748cca56109df6ccd237440b7c88b" gracePeriod=30 Sep 11 14:36:37.421496 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.421439 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:36:37.480152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.480122 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x62d5\" (UniqueName: \"kubernetes.io/projected/cd2fcdf5-39c6-450e-975f-850f88e4ba29-kube-api-access-x62d5\") pod \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " Sep 11 14:36:37.480304 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.480188 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd2fcdf5-39c6-450e-975f-850f88e4ba29-maas-api-tls\") pod \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " Sep 11 14:36:37.480304 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.480243 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd2fcdf5-39c6-450e-975f-850f88e4ba29-openshift-service-ca\") pod \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " Sep 11 14:36:37.480304 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.480258 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd2fcdf5-39c6-450e-975f-850f88e4ba29-metrics-certs\") pod \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\" (UID: \"cd2fcdf5-39c6-450e-975f-850f88e4ba29\") " Sep 11 14:36:37.480577 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.480546 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2fcdf5-39c6-450e-975f-850f88e4ba29-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd2fcdf5-39c6-450e-975f-850f88e4ba29" (UID: "cd2fcdf5-39c6-450e-975f-850f88e4ba29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:36:37.482302 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.482273 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd2fcdf5-39c6-450e-975f-850f88e4ba29-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd2fcdf5-39c6-450e-975f-850f88e4ba29" (UID: "cd2fcdf5-39c6-450e-975f-850f88e4ba29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:36:37.482421 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.482348 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd2fcdf5-39c6-450e-975f-850f88e4ba29-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cd2fcdf5-39c6-450e-975f-850f88e4ba29" (UID: "cd2fcdf5-39c6-450e-975f-850f88e4ba29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:36:37.482421 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.482377 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2fcdf5-39c6-450e-975f-850f88e4ba29-kube-api-access-x62d5" (OuterVolumeSpecName: "kube-api-access-x62d5") pod "cd2fcdf5-39c6-450e-975f-850f88e4ba29" (UID: "cd2fcdf5-39c6-450e-975f-850f88e4ba29"). InnerVolumeSpecName "kube-api-access-x62d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:36:37.581324 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.581296 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd2fcdf5-39c6-450e-975f-850f88e4ba29-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:36:37.581324 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.581324 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd2fcdf5-39c6-450e-975f-850f88e4ba29-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:36:37.581517 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.581337 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x62d5\" (UniqueName: \"kubernetes.io/projected/cd2fcdf5-39c6-450e-975f-850f88e4ba29-kube-api-access-x62d5\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:36:37.581517 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.581349 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd2fcdf5-39c6-450e-975f-850f88e4ba29-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:36:37.682572 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.682476 2578 generic.go:356] "Generic (PLEG): container finished" podID="cd2fcdf5-39c6-450e-975f-850f88e4ba29" containerID="a56482cc5eece74156e7a42d4186482ada7748cca56109df6ccd237440b7c88b" exitCode=0 Sep 11 14:36:37.682572 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.682526 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" Sep 11 14:36:37.682845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.682574 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" event={"ID":"cd2fcdf5-39c6-450e-975f-850f88e4ba29","Type":"ContainerDied","Data":"a56482cc5eece74156e7a42d4186482ada7748cca56109df6ccd237440b7c88b"} Sep 11 14:36:37.682845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.682640 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq" event={"ID":"cd2fcdf5-39c6-450e-975f-850f88e4ba29","Type":"ContainerDied","Data":"eafbe285346a5cec6355b8d762003af63729a8f801546d7473bb6278f06ab604"} Sep 11 14:36:37.682845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.682673 2578 scope.go:117] "RemoveContainer" containerID="a56482cc5eece74156e7a42d4186482ada7748cca56109df6ccd237440b7c88b" Sep 11 14:36:37.693181 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.693164 2578 scope.go:117] "RemoveContainer" containerID="a56482cc5eece74156e7a42d4186482ada7748cca56109df6ccd237440b7c88b" Sep 11 14:36:37.693451 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:36:37.693434 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56482cc5eece74156e7a42d4186482ada7748cca56109df6ccd237440b7c88b\": container with ID starting with a56482cc5eece74156e7a42d4186482ada7748cca56109df6ccd237440b7c88b not found: ID does not exist" containerID="a56482cc5eece74156e7a42d4186482ada7748cca56109df6ccd237440b7c88b" Sep 11 14:36:37.693525 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.693458 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56482cc5eece74156e7a42d4186482ada7748cca56109df6ccd237440b7c88b"} err="failed to get container status \"a56482cc5eece74156e7a42d4186482ada7748cca56109df6ccd237440b7c88b\": rpc error: code = NotFound desc = could not find container \"a56482cc5eece74156e7a42d4186482ada7748cca56109df6ccd237440b7c88b\": container with ID starting with a56482cc5eece74156e7a42d4186482ada7748cca56109df6ccd237440b7c88b not found: ID does not exist" Sep 11 14:36:37.712119 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.712094 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq"] Sep 11 14:36:37.718205 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:37.718176 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c3340176-6bd5c56c8d-s76kq"] Sep 11 14:36:38.464122 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:38.464088 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd2fcdf5-39c6-450e-975f-850f88e4ba29" path="/var/lib/kubelet/pods/cd2fcdf5-39c6-450e-975f-850f88e4ba29/volumes" Sep 11 14:36:45.462528 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:45.462493 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:36:47.040436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:47.040404 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg"] Sep 11 14:36:47.040849 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:47.040593 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" containerID="cri-o://60a8c38201ccc85fc22aecae792278e190a1203e8e9eb9e0fd26b252c3134603" gracePeriod=30 Sep 11 14:36:52.294270 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.294242 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:36:52.403483 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.403454 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-data\") pod \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " Sep 11 14:36:52.403668 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.403506 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-workload-socket\") pod \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " Sep 11 14:36:52.403668 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.403535 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-envoy\") pod \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " Sep 11 14:36:52.403668 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.403610 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istiod-ca-cert\") pod \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " Sep 11 14:36:52.403668 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.403640 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-token\") pod \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " Sep 11 14:36:52.403668 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.403664 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-credential-socket\") pod \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " Sep 11 14:36:52.403979 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.403704 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-workload-certs\") pod \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " Sep 11 14:36:52.403979 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.403753 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6vl7\" (UniqueName: \"kubernetes.io/projected/a7cb9caa-f3a6-499a-8213-09c8d5b12903-kube-api-access-k6vl7\") pod \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " Sep 11 14:36:52.403979 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.403756 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7cb9caa-f3a6-499a-8213-09c8d5b12903" (UID: "a7cb9caa-f3a6-499a-8213-09c8d5b12903"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:52.403979 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.403756 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7cb9caa-f3a6-499a-8213-09c8d5b12903" (UID: "a7cb9caa-f3a6-499a-8213-09c8d5b12903"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:52.403979 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.403785 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-podinfo\") pod \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\" (UID: \"a7cb9caa-f3a6-499a-8213-09c8d5b12903\") " Sep 11 14:36:52.404262 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.403984 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7cb9caa-f3a6-499a-8213-09c8d5b12903" (UID: "a7cb9caa-f3a6-499a-8213-09c8d5b12903"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:52.404262 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.404054 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7cb9caa-f3a6-499a-8213-09c8d5b12903" (UID: "a7cb9caa-f3a6-499a-8213-09c8d5b12903"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:36:52.404262 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.404132 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-data\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:36:52.404262 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.404152 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-workload-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:36:52.404262 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.404164 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-credential-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:36:52.404262 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.404130 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7cb9caa-f3a6-499a-8213-09c8d5b12903" (UID: "a7cb9caa-f3a6-499a-8213-09c8d5b12903"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:52.406114 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.406080 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7cb9caa-f3a6-499a-8213-09c8d5b12903" (UID: "a7cb9caa-f3a6-499a-8213-09c8d5b12903"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:36:52.406276 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.406219 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7cb9caa-f3a6-499a-8213-09c8d5b12903-kube-api-access-k6vl7" (OuterVolumeSpecName: "kube-api-access-k6vl7") pod "a7cb9caa-f3a6-499a-8213-09c8d5b12903" (UID: "a7cb9caa-f3a6-499a-8213-09c8d5b12903"). InnerVolumeSpecName "kube-api-access-k6vl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:36:52.406348 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.406310 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7cb9caa-f3a6-499a-8213-09c8d5b12903" (UID: "a7cb9caa-f3a6-499a-8213-09c8d5b12903"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:36:52.406348 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.406336 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7cb9caa-f3a6-499a-8213-09c8d5b12903" (UID: "a7cb9caa-f3a6-499a-8213-09c8d5b12903"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:52.505109 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.505081 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-envoy\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:36:52.505109 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.505105 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istiod-ca-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:36:52.505109 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.505114 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-token\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:36:52.505290 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.505125 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7cb9caa-f3a6-499a-8213-09c8d5b12903-workload-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:36:52.505290 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.505133 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6vl7\" (UniqueName: \"kubernetes.io/projected/a7cb9caa-f3a6-499a-8213-09c8d5b12903-kube-api-access-k6vl7\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:36:52.505290 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.505142 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7cb9caa-f3a6-499a-8213-09c8d5b12903-istio-podinfo\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:36:52.773270 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.773188 2578 generic.go:356] "Generic (PLEG): container finished" podID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerID="60a8c38201ccc85fc22aecae792278e190a1203e8e9eb9e0fd26b252c3134603" exitCode=0 Sep 11 14:36:52.773270 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.773249 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" Sep 11 14:36:52.773446 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.773240 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" event={"ID":"a7cb9caa-f3a6-499a-8213-09c8d5b12903","Type":"ContainerDied","Data":"60a8c38201ccc85fc22aecae792278e190a1203e8e9eb9e0fd26b252c3134603"} Sep 11 14:36:52.773446 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.773371 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg" event={"ID":"a7cb9caa-f3a6-499a-8213-09c8d5b12903","Type":"ContainerDied","Data":"1506f26f40bdd6f1643452c4516e8334b23fdd19e534f12aadae328f02c75802"} Sep 11 14:36:52.773446 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.773400 2578 scope.go:117] "RemoveContainer" containerID="60a8c38201ccc85fc22aecae792278e190a1203e8e9eb9e0fd26b252c3134603" Sep 11 14:36:52.783963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.783946 2578 scope.go:117] "RemoveContainer" containerID="9956364e34aebab4b2089f0125ec5afca11e34929330a9ff4ca6779db7dd002f" Sep 11 14:36:52.792290 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.792274 2578 scope.go:117] "RemoveContainer" containerID="60a8c38201ccc85fc22aecae792278e190a1203e8e9eb9e0fd26b252c3134603" Sep 11 14:36:52.792576 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:36:52.792536 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a8c38201ccc85fc22aecae792278e190a1203e8e9eb9e0fd26b252c3134603\": container with ID starting with 60a8c38201ccc85fc22aecae792278e190a1203e8e9eb9e0fd26b252c3134603 not found: ID does not exist" containerID="60a8c38201ccc85fc22aecae792278e190a1203e8e9eb9e0fd26b252c3134603" Sep 11 14:36:52.792633 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.792584 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a8c38201ccc85fc22aecae792278e190a1203e8e9eb9e0fd26b252c3134603"} err="failed to get container status \"60a8c38201ccc85fc22aecae792278e190a1203e8e9eb9e0fd26b252c3134603\": rpc error: code = NotFound desc = could not find container \"60a8c38201ccc85fc22aecae792278e190a1203e8e9eb9e0fd26b252c3134603\": container with ID starting with 60a8c38201ccc85fc22aecae792278e190a1203e8e9eb9e0fd26b252c3134603 not found: ID does not exist" Sep 11 14:36:52.792633 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.792599 2578 scope.go:117] "RemoveContainer" containerID="9956364e34aebab4b2089f0125ec5afca11e34929330a9ff4ca6779db7dd002f" Sep 11 14:36:52.792852 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:36:52.792833 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9956364e34aebab4b2089f0125ec5afca11e34929330a9ff4ca6779db7dd002f\": container with ID starting with 9956364e34aebab4b2089f0125ec5afca11e34929330a9ff4ca6779db7dd002f not found: ID does not exist" containerID="9956364e34aebab4b2089f0125ec5afca11e34929330a9ff4ca6779db7dd002f" Sep 11 14:36:52.792901 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.792857 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9956364e34aebab4b2089f0125ec5afca11e34929330a9ff4ca6779db7dd002f"} err="failed to get container status \"9956364e34aebab4b2089f0125ec5afca11e34929330a9ff4ca6779db7dd002f\": rpc error: code = NotFound desc = could not find container \"9956364e34aebab4b2089f0125ec5afca11e34929330a9ff4ca6779db7dd002f\": container with ID starting with 9956364e34aebab4b2089f0125ec5afca11e34929330a9ff4ca6779db7dd002f not found: ID does not exist" Sep 11 14:36:52.811829 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.811804 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg"] Sep 11 14:36:52.827609 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:52.827589 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c3340176-openshift-default-7c4c495788-4fplg"] Sep 11 14:36:54.464656 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:54.464621 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" path="/var/lib/kubelet/pods/a7cb9caa-f3a6-499a-8213-09c8d5b12903/volumes" Sep 11 14:36:57.840099 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:36:57.840061 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:37:06.092742 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.092690 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv"] Sep 11 14:37:06.093229 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.093199 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" Sep 11 14:37:06.093229 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.093215 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" Sep 11 14:37:06.093229 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.093224 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd2fcdf5-39c6-450e-975f-850f88e4ba29" containerName="maas-api" Sep 11 14:37:06.093229 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.093230 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd2fcdf5-39c6-450e-975f-850f88e4ba29" containerName="maas-api" Sep 11 14:37:06.093367 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.093298 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd2fcdf5-39c6-450e-975f-850f88e4ba29" containerName="maas-api" Sep 11 14:37:06.093367 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.093309 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" Sep 11 14:37:06.098025 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.098008 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.101193 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.101164 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 14:37:06.101324 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.101289 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-958398-openshift-default-dockercfg-r4m7b\"" Sep 11 14:37:06.121548 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.121525 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv"] Sep 11 14:37:06.221277 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.221243 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e553792-1de7-4dd9-b43f-d22976b0d94f-istiod-ca-cert\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.221442 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.221283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-workload-certs\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.221442 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.221308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-token\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.221442 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.221399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-credential-socket\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.221548 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.221442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-podinfo\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.221548 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.221461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8kdj\" (UniqueName: \"kubernetes.io/projected/1e553792-1de7-4dd9-b43f-d22976b0d94f-kube-api-access-l8kdj\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.221548 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.221510 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-workload-socket\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.221548 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.221530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-envoy\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.221548 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.221546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-data\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.322440 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.322405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-envoy\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.322440 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.322439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-data\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.322687 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.322484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e553792-1de7-4dd9-b43f-d22976b0d94f-istiod-ca-cert\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.322687 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.322503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-workload-certs\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.322687 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.322522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-token\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.322687 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.322566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-credential-socket\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.322957 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.322717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-podinfo\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.322957 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.322788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8kdj\" (UniqueName: \"kubernetes.io/projected/1e553792-1de7-4dd9-b43f-d22976b0d94f-kube-api-access-l8kdj\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.322957 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.322866 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-workload-socket\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.323109 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.322967 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-data\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.323109 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.323004 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-workload-certs\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.323109 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.323039 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-credential-socket\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.323268 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.323198 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-workload-socket\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.323356 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.323335 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e553792-1de7-4dd9-b43f-d22976b0d94f-istiod-ca-cert\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.324785 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.324764 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-envoy\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.325124 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.325109 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-podinfo\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.346222 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.346160 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-token\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.346685 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.346667 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8kdj\" (UniqueName: \"kubernetes.io/projected/1e553792-1de7-4dd9-b43f-d22976b0d94f-kube-api-access-l8kdj\") pod \"e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.409432 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.409397 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:06.734539 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.734499 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv"] Sep 11 14:37:06.736879 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:37:06.736852 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e553792_1de7_4dd9_b43f_d22976b0d94f.slice/crio-87944d8a952ff0b7d9ea1ed71cf57073222069026c3575b316d48732d032312e WatchSource:0}: Error finding container 87944d8a952ff0b7d9ea1ed71cf57073222069026c3575b316d48732d032312e: Status 404 returned error can't find the container with id 87944d8a952ff0b7d9ea1ed71cf57073222069026c3575b316d48732d032312e Sep 11 14:37:06.739054 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.739017 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:37:06.739123 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.739086 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:37:06.739123 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.739113 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:37:06.864071 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.864030 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" event={"ID":"1e553792-1de7-4dd9-b43f-d22976b0d94f","Type":"ContainerStarted","Data":"e0f2a81b3a4e3ae4994bbdb9c1f19838399dbce115fef879e89628b7bc1e5643"} Sep 11 14:37:06.864257 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.864134 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" event={"ID":"1e553792-1de7-4dd9-b43f-d22976b0d94f","Type":"ContainerStarted","Data":"87944d8a952ff0b7d9ea1ed71cf57073222069026c3575b316d48732d032312e"} Sep 11 14:37:06.896934 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:06.896877 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" podStartSLOduration=0.896862626 podStartE2EDuration="896.862626ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:06.896678747 +0000 UTC m=+1314.982135107" watchObservedRunningTime="2026-09-11 14:37:06.896862626 +0000 UTC m=+1314.982318986" Sep 11 14:37:07.409779 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:07.409746 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:07.414576 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:07.414547 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:07.869821 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:07.869791 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:07.870800 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:07.870779 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:12.564422 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.564383 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc"] Sep 11 14:37:12.564867 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.564834 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" Sep 11 14:37:12.564867 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.564847 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" Sep 11 14:37:12.564977 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.564911 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7cb9caa-f3a6-499a-8213-09c8d5b12903" containerName="istio-proxy" Sep 11 14:37:12.569425 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.569403 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:12.572209 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.572186 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-958398-serving-cert\"" Sep 11 14:37:12.585462 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.585433 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc"] Sep 11 14:37:12.677158 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.677132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:12.677320 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.677181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8k5f\" (UniqueName: \"kubernetes.io/projected/b940871f-8c89-477c-9755-6f3f54e096df-kube-api-access-r8k5f\") pod \"maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:12.677320 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.677293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b940871f-8c89-477c-9755-6f3f54e096df-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:12.677320 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.677310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:12.778517 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.778435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b940871f-8c89-477c-9755-6f3f54e096df-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:12.778517 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.778474 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:12.778775 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.778524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:12.778775 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.778571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8k5f\" (UniqueName: \"kubernetes.io/projected/b940871f-8c89-477c-9755-6f3f54e096df-kube-api-access-r8k5f\") pod \"maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:12.778775 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:37:12.778664 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-serving-cert: secret "maas-api-e2e-shared-a-w2-958398-serving-cert" not found Sep 11 14:37:12.778944 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:37:12.778790 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-maas-api-tls podName:b940871f-8c89-477c-9755-6f3f54e096df nodeName:}" failed. No retries permitted until 2026-09-11 14:37:13.278767212 +0000 UTC m=+1321.364223563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-maas-api-tls") pod "maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" (UID: "b940871f-8c89-477c-9755-6f3f54e096df") : secret "maas-api-e2e-shared-a-w2-958398-serving-cert" not found Sep 11 14:37:12.779200 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.779175 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b940871f-8c89-477c-9755-6f3f54e096df-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:12.780990 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.780968 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:12.790614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:12.790590 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8k5f\" (UniqueName: \"kubernetes.io/projected/b940871f-8c89-477c-9755-6f3f54e096df-kube-api-access-r8k5f\") pod \"maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:13.283005 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:13.282974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:13.285710 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:13.285685 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:13.479888 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:13.479853 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:13.770523 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:13.770498 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc"] Sep 11 14:37:13.771713 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:37:13.771683 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb940871f_8c89_477c_9755_6f3f54e096df.slice/crio-3be721ce33420c6dfe399e5ee424744fa30339f0cca1f81973a73a168f80e318 WatchSource:0}: Error finding container 3be721ce33420c6dfe399e5ee424744fa30339f0cca1f81973a73a168f80e318: Status 404 returned error can't find the container with id 3be721ce33420c6dfe399e5ee424744fa30339f0cca1f81973a73a168f80e318 Sep 11 14:37:13.910803 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:13.910777 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" event={"ID":"b940871f-8c89-477c-9755-6f3f54e096df","Type":"ContainerStarted","Data":"3be721ce33420c6dfe399e5ee424744fa30339f0cca1f81973a73a168f80e318"} Sep 11 14:37:14.917980 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:14.917935 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" event={"ID":"b940871f-8c89-477c-9755-6f3f54e096df","Type":"ContainerStarted","Data":"426d97bb91587a8781eed824c4ac2c1200349985cd0756c24caf2c13095c8adc"} Sep 11 14:37:14.918344 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:14.918085 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:14.963641 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:14.963583 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" podStartSLOduration=2.466701587 podStartE2EDuration="2.96356701s" totalImagesPullingTime="496.865423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:14.961126988 +0000 UTC m=+1323.046583350" watchObservedRunningTime="2026-09-11 14:37:14.96356701 +0000 UTC m=+1323.049023370" Sep 11 14:37:16.548045 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:16.548007 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:37:20.929900 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:20.929871 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:37:28.015338 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:28.015303 2578 generic.go:356] "Generic (PLEG): container finished" podID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerID="e0f2a81b3a4e3ae4994bbdb9c1f19838399dbce115fef879e89628b7bc1e5643" exitCode=0 Sep 11 14:37:28.015785 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:28.015361 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" event={"ID":"1e553792-1de7-4dd9-b43f-d22976b0d94f","Type":"ContainerDied","Data":"e0f2a81b3a4e3ae4994bbdb9c1f19838399dbce115fef879e89628b7bc1e5643"} Sep 11 14:37:28.015785 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:28.015716 2578 scope.go:117] "RemoveContainer" containerID="e0f2a81b3a4e3ae4994bbdb9c1f19838399dbce115fef879e89628b7bc1e5643" Sep 11 14:37:28.016714 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:28.016688 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:37:28.016801 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:28.016777 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:37:28.016839 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:28.016829 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:37:29.023254 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:29.023219 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" event={"ID":"1e553792-1de7-4dd9-b43f-d22976b0d94f","Type":"ContainerStarted","Data":"66d0dbb53ef661a64d71ace54d3679239047d76f5ee93ef058ef91b7fb614f97"} Sep 11 14:37:29.410295 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:29.410257 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:29.410473 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:29.410456 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 11 14:37:29.410523 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:29.410507 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 11 14:37:30.409883 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:30.409843 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 11 14:37:30.410274 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:30.409908 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 11 14:37:31.410430 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:31.410394 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 11 14:37:31.411004 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:31.410464 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 11 14:37:32.410081 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:32.410043 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 11 14:37:32.410256 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:32.410099 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 11 14:37:33.409945 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:33.409909 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 11 14:37:33.410307 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:33.409968 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 11 14:37:34.410545 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:34.410507 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 11 14:37:34.410979 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:34.410559 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 11 14:37:35.413881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:35.413785 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:35.414316 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:35.414099 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:35.414773 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:35.414753 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:37:36.354292 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:36.354255 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:37:40.041029 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.040997 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5"] Sep 11 14:37:40.044144 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.044127 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.046975 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.046954 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-455f2e-openshift-default-dockercfg-5kj4j\"" Sep 11 14:37:40.061616 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.061592 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5"] Sep 11 14:37:40.153697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.153654 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-credential-socket\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.153697 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.153700 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istiod-ca-cert\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.153968 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.153738 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-podinfo\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.153968 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.153862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-workload-certs\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.153968 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.153911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-token\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.153968 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.153943 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-workload-socket\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.154186 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.153970 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-envoy\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.154186 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.154028 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86zw\" (UniqueName: \"kubernetes.io/projected/e7001d7a-af65-4b81-9c1f-4b31a60a150a-kube-api-access-t86zw\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.154186 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.154068 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-data\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.255539 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.255498 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-credential-socket\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.255768 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.255548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istiod-ca-cert\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.255768 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.255583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-podinfo\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.255768 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.255624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-workload-certs\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.255768 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.255695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-token\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.256014 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.255782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-workload-socket\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.256014 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.255914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-envoy\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.256014 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.255964 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-credential-socket\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.256174 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.256042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86zw\" (UniqueName: \"kubernetes.io/projected/e7001d7a-af65-4b81-9c1f-4b31a60a150a-kube-api-access-t86zw\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.256174 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.256070 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-workload-socket\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.256174 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.256107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-data\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.256410 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.256386 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-workload-certs\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.256476 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.256439 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-data\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.256677 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.256660 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istiod-ca-cert\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.258239 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.258201 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-envoy\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.258430 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.258411 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-podinfo\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.267180 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.267155 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-token\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.267374 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.267357 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86zw\" (UniqueName: \"kubernetes.io/projected/e7001d7a-af65-4b81-9c1f-4b31a60a150a-kube-api-access-t86zw\") pod \"e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.355330 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.355297 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:40.640715 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.640690 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5"] Sep 11 14:37:40.642003 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:37:40.641970 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7001d7a_af65_4b81_9c1f_4b31a60a150a.slice/crio-f5621bb23baafdf6931145d254eb4091fc6946d2136c31acf389d90a49467f48 WatchSource:0}: Error finding container f5621bb23baafdf6931145d254eb4091fc6946d2136c31acf389d90a49467f48: Status 404 returned error can't find the container with id f5621bb23baafdf6931145d254eb4091fc6946d2136c31acf389d90a49467f48 Sep 11 14:37:40.644337 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.644303 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:37:40.644438 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.644382 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:37:40.644438 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:40.644430 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:37:41.096616 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:41.096530 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" event={"ID":"e7001d7a-af65-4b81-9c1f-4b31a60a150a","Type":"ContainerStarted","Data":"4ba045327d5f7dc1b214e0fc036113a47ab9758da9d1c2c5245ec6122adacb6d"} Sep 11 14:37:41.096616 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:41.096585 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" event={"ID":"e7001d7a-af65-4b81-9c1f-4b31a60a150a","Type":"ContainerStarted","Data":"f5621bb23baafdf6931145d254eb4091fc6946d2136c31acf389d90a49467f48"} Sep 11 14:37:41.131153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:41.131089 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" podStartSLOduration=1.131070274 podStartE2EDuration="1.131070274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:41.121944812 +0000 UTC m=+1349.207401173" watchObservedRunningTime="2026-09-11 14:37:41.131070274 +0000 UTC m=+1349.216526710" Sep 11 14:37:41.356256 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:41.356222 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:41.361091 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:41.361067 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:42.103308 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:42.103271 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:42.104276 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:42.104251 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:37:47.767672 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:47.767640 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw"] Sep 11 14:37:47.772061 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:47.772039 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:47.775210 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:47.775186 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-455f2e-serving-cert\"" Sep 11 14:37:47.796153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:47.796123 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw"] Sep 11 14:37:47.927625 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:47.927586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlq76\" (UniqueName: \"kubernetes.io/projected/f5f0985b-b8f7-4140-9870-5a481665636b-kube-api-access-hlq76\") pod \"maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:47.927625 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:47.927627 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:47.927933 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:47.927770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f0985b-b8f7-4140-9870-5a481665636b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:47.927933 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:47.927809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:48.028766 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:48.028664 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlq76\" (UniqueName: \"kubernetes.io/projected/f5f0985b-b8f7-4140-9870-5a481665636b-kube-api-access-hlq76\") pod \"maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:48.028766 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:48.028701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:48.029003 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:48.028771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f0985b-b8f7-4140-9870-5a481665636b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:48.029003 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:48.028788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:48.029096 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:37:48.029022 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-serving-cert: secret "maas-api-e2e-shared-b-w2-455f2e-serving-cert" not found Sep 11 14:37:48.029134 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:37:48.029114 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-maas-api-tls podName:f5f0985b-b8f7-4140-9870-5a481665636b nodeName:}" failed. No retries permitted until 2026-09-11 14:37:48.529096712 +0000 UTC m=+1356.614553050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-maas-api-tls") pod "maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" (UID: "f5f0985b-b8f7-4140-9870-5a481665636b") : secret "maas-api-e2e-shared-b-w2-455f2e-serving-cert" not found Sep 11 14:37:48.029411 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:48.029384 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f0985b-b8f7-4140-9870-5a481665636b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:48.031155 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:48.031133 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:48.047176 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:48.047153 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlq76\" (UniqueName: \"kubernetes.io/projected/f5f0985b-b8f7-4140-9870-5a481665636b-kube-api-access-hlq76\") pod \"maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:48.534724 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:48.534690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:48.537482 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:48.537458 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:48.683046 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:48.683014 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:48.979193 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:48.979163 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw"] Sep 11 14:37:48.980077 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:37:48.980050 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f0985b_b8f7_4140_9870_5a481665636b.slice/crio-dcf509b05e375cb0c9645d10ac1222b4981f72a0b056f08d57a8f7a286bbc092 WatchSource:0}: Error finding container dcf509b05e375cb0c9645d10ac1222b4981f72a0b056f08d57a8f7a286bbc092: Status 404 returned error can't find the container with id dcf509b05e375cb0c9645d10ac1222b4981f72a0b056f08d57a8f7a286bbc092 Sep 11 14:37:49.145918 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:49.145876 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" event={"ID":"f5f0985b-b8f7-4140-9870-5a481665636b","Type":"ContainerStarted","Data":"dcf509b05e375cb0c9645d10ac1222b4981f72a0b056f08d57a8f7a286bbc092"} Sep 11 14:37:50.154941 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:50.154908 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" event={"ID":"f5f0985b-b8f7-4140-9870-5a481665636b","Type":"ContainerStarted","Data":"532d1f44cfb09070e2e5ddd86f04e66f5caeb1d1887ce25532cb6eb98aeee717"} Sep 11 14:37:50.155342 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:50.155134 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:37:50.192534 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:50.192485 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" podStartSLOduration=2.605871226 podStartE2EDuration="3.192469568s" totalImagesPullingTime="586.598342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:50.191781848 +0000 UTC m=+1358.277238208" watchObservedRunningTime="2026-09-11 14:37:50.192469568 +0000 UTC m=+1358.277925928" Sep 11 14:37:52.946999 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:52.946963 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:37:56.172828 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:37:56.172066 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:38:03.247656 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:03.247621 2578 generic.go:356] "Generic (PLEG): container finished" podID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerID="4ba045327d5f7dc1b214e0fc036113a47ab9758da9d1c2c5245ec6122adacb6d" exitCode=0 Sep 11 14:38:03.248199 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:03.247700 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" event={"ID":"e7001d7a-af65-4b81-9c1f-4b31a60a150a","Type":"ContainerDied","Data":"4ba045327d5f7dc1b214e0fc036113a47ab9758da9d1c2c5245ec6122adacb6d"} Sep 11 14:38:03.248305 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:03.248285 2578 scope.go:117] "RemoveContainer" containerID="4ba045327d5f7dc1b214e0fc036113a47ab9758da9d1c2c5245ec6122adacb6d" Sep 11 14:38:03.249478 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:03.249446 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:03.249600 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:03.249580 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:03.249666 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:03.249633 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:04.257603 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:04.257568 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" event={"ID":"e7001d7a-af65-4b81-9c1f-4b31a60a150a","Type":"ContainerStarted","Data":"027edfc9145a20c08f5a2a2da6d1b2f578c86719186f9b241bf8ff8eb915d5ee"} Sep 11 14:38:04.355962 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:04.355922 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:38:04.356168 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:04.356059 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Sep 11 14:38:04.356168 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:04.356105 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Sep 11 14:38:05.356616 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:05.356571 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Sep 11 14:38:05.357121 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:05.356645 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Sep 11 14:38:06.356198 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:06.356153 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Sep 11 14:38:06.356406 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:06.356237 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Sep 11 14:38:07.356025 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:07.355982 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Sep 11 14:38:07.356522 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:07.356057 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Sep 11 14:38:08.356661 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:08.356622 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Sep 11 14:38:08.357281 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:08.357251 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Sep 11 14:38:09.356198 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:09.356161 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Sep 11 14:38:09.356492 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:09.356458 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Sep 11 14:38:10.360496 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:10.360464 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:38:10.360993 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:10.360853 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:38:10.361628 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:10.361609 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:38:13.446744 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:13.446691 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:38:14.201815 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:14.201778 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8013-0fedfe"] Sep 11 14:38:14.204330 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:14.204311 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-0fedfe" Sep 11 14:38:14.207273 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:14.207251 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7pn49\"" Sep 11 14:38:14.217424 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:14.217401 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8013-0fedfe"] Sep 11 14:38:14.281842 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:14.281816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf77f\" (UniqueName: \"kubernetes.io/projected/cd6d5873-4af1-4e84-b9b7-b3a4998543e7-kube-api-access-rf77f\") pod \"test-curl-8013-0fedfe\" (UID: \"cd6d5873-4af1-4e84-b9b7-b3a4998543e7\") " pod="opendatahub/test-curl-8013-0fedfe" Sep 11 14:38:14.383259 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:14.383224 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf77f\" (UniqueName: \"kubernetes.io/projected/cd6d5873-4af1-4e84-b9b7-b3a4998543e7-kube-api-access-rf77f\") pod \"test-curl-8013-0fedfe\" (UID: \"cd6d5873-4af1-4e84-b9b7-b3a4998543e7\") " pod="opendatahub/test-curl-8013-0fedfe" Sep 11 14:38:14.395752 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:14.395702 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf77f\" (UniqueName: \"kubernetes.io/projected/cd6d5873-4af1-4e84-b9b7-b3a4998543e7-kube-api-access-rf77f\") pod \"test-curl-8013-0fedfe\" (UID: \"cd6d5873-4af1-4e84-b9b7-b3a4998543e7\") " pod="opendatahub/test-curl-8013-0fedfe" Sep 11 14:38:14.513509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:14.513437 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-0fedfe" Sep 11 14:38:14.795968 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:14.795941 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8013-0fedfe"] Sep 11 14:38:14.796786 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:38:14.796758 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd6d5873_4af1_4e84_b9b7_b3a4998543e7.slice/crio-1a98ab5016b55541fee1a2e6cee83c6ad02552864537acc04155a8a8efdd1a33 WatchSource:0}: Error finding container 1a98ab5016b55541fee1a2e6cee83c6ad02552864537acc04155a8a8efdd1a33: Status 404 returned error can't find the container with id 1a98ab5016b55541fee1a2e6cee83c6ad02552864537acc04155a8a8efdd1a33 Sep 11 14:38:15.334657 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:15.334625 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8013-0fedfe" event={"ID":"cd6d5873-4af1-4e84-b9b7-b3a4998543e7","Type":"ContainerStarted","Data":"ff390c21ccbcb5d1084d1aafd54ba510b6dda02029069c93dbe115f0502180ff"} Sep 11 14:38:15.334879 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:15.334671 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8013-0fedfe" event={"ID":"cd6d5873-4af1-4e84-b9b7-b3a4998543e7","Type":"ContainerStarted","Data":"1a98ab5016b55541fee1a2e6cee83c6ad02552864537acc04155a8a8efdd1a33"} Sep 11 14:38:15.356957 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:15.356907 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8013-0fedfe" podStartSLOduration=1.356889567 podStartE2EDuration="1.356889567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:15.356657907 +0000 UTC m=+1383.442114267" watchObservedRunningTime="2026-09-11 14:38:15.356889567 +0000 UTC m=+1383.442345927" Sep 11 14:38:16.077500 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:16.077451 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8013-0fedfe"] Sep 11 14:38:16.087495 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:16.087420 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8013-0fedfe"] Sep 11 14:38:16.224667 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:16.224626 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8013-5140b5"] Sep 11 14:38:16.225135 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:16.225121 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd6d5873-4af1-4e84-b9b7-b3a4998543e7" containerName="test-curl-8013-0fedfe" Sep 11 14:38:16.225182 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:16.225136 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd6d5873-4af1-4e84-b9b7-b3a4998543e7" containerName="test-curl-8013-0fedfe" Sep 11 14:38:16.225226 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:16.225198 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd6d5873-4af1-4e84-b9b7-b3a4998543e7" containerName="test-curl-8013-0fedfe" Sep 11 14:38:16.227336 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:16.227320 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-5140b5" Sep 11 14:38:16.249030 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:16.249001 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8013-5140b5"] Sep 11 14:38:16.300934 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:16.300900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9jrz\" (UniqueName: \"kubernetes.io/projected/7b573e1d-fe19-4771-978e-650b51180f9a-kube-api-access-v9jrz\") pod \"test-curl-8013-5140b5\" (UID: \"7b573e1d-fe19-4771-978e-650b51180f9a\") " pod="opendatahub/test-curl-8013-5140b5" Sep 11 14:38:16.340718 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:16.340630 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8013-0fedfe" container="test-curl-8013-0fedfe" Sep 11 14:38:16.401909 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:16.401864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9jrz\" (UniqueName: \"kubernetes.io/projected/7b573e1d-fe19-4771-978e-650b51180f9a-kube-api-access-v9jrz\") pod \"test-curl-8013-5140b5\" (UID: \"7b573e1d-fe19-4771-978e-650b51180f9a\") " pod="opendatahub/test-curl-8013-5140b5" Sep 11 14:38:16.416545 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:16.416512 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9jrz\" (UniqueName: \"kubernetes.io/projected/7b573e1d-fe19-4771-978e-650b51180f9a-kube-api-access-v9jrz\") pod \"test-curl-8013-5140b5\" (UID: \"7b573e1d-fe19-4771-978e-650b51180f9a\") " pod="opendatahub/test-curl-8013-5140b5" Sep 11 14:38:16.535789 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:16.535756 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-5140b5" Sep 11 14:38:16.825817 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:16.825788 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8013-5140b5"] Sep 11 14:38:16.826326 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:38:16.826296 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b573e1d_fe19_4771_978e_650b51180f9a.slice/crio-60c28833dc5f2cd1a3278896f0c695b5762cca352fae903e23a9e1cebe932da4 WatchSource:0}: Error finding container 60c28833dc5f2cd1a3278896f0c695b5762cca352fae903e23a9e1cebe932da4: Status 404 returned error can't find the container with id 60c28833dc5f2cd1a3278896f0c695b5762cca352fae903e23a9e1cebe932da4 Sep 11 14:38:17.348172 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:17.348138 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8013-5140b5" event={"ID":"7b573e1d-fe19-4771-978e-650b51180f9a","Type":"ContainerStarted","Data":"d8c5863c38eaea437bf0c37f4ec0033fb6c616691d07130329b0685147ee64bd"} Sep 11 14:38:17.348579 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:17.348181 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8013-5140b5" event={"ID":"7b573e1d-fe19-4771-978e-650b51180f9a","Type":"ContainerStarted","Data":"60c28833dc5f2cd1a3278896f0c695b5762cca352fae903e23a9e1cebe932da4"} Sep 11 14:38:17.348579 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:17.348174 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8013-0fedfe" podUID="cd6d5873-4af1-4e84-b9b7-b3a4998543e7" containerName="test-curl-8013-0fedfe" containerID="cri-o://ff390c21ccbcb5d1084d1aafd54ba510b6dda02029069c93dbe115f0502180ff" gracePeriod=2 Sep 11 14:38:17.385881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:17.385825 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8013-5140b5" podStartSLOduration=1.385807175 podStartE2EDuration="1.385807175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:17.385509045 +0000 UTC m=+1385.470965405" watchObservedRunningTime="2026-09-11 14:38:17.385807175 +0000 UTC m=+1385.471263533" Sep 11 14:38:18.107601 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:18.107570 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8013-5140b5"] Sep 11 14:38:18.117155 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:18.117125 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8013-5140b5"] Sep 11 14:38:18.262708 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:18.262675 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8013-0c983f"] Sep 11 14:38:18.263123 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:18.263110 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b573e1d-fe19-4771-978e-650b51180f9a" containerName="test-curl-8013-5140b5" Sep 11 14:38:18.263170 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:18.263125 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b573e1d-fe19-4771-978e-650b51180f9a" containerName="test-curl-8013-5140b5" Sep 11 14:38:18.263214 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:18.263205 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b573e1d-fe19-4771-978e-650b51180f9a" containerName="test-curl-8013-5140b5" Sep 11 14:38:18.265196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:18.265180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-0c983f" Sep 11 14:38:18.280376 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:18.280348 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8013-0c983f"] Sep 11 14:38:18.354696 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:18.354667 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8013-5140b5" container="test-curl-8013-5140b5" Sep 11 14:38:18.422040 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:18.421962 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68cd4\" (UniqueName: \"kubernetes.io/projected/a1d308d4-c276-4a36-9ace-de849bc7c6d2-kube-api-access-68cd4\") pod \"test-curl-8013-0c983f\" (UID: \"a1d308d4-c276-4a36-9ace-de849bc7c6d2\") " pod="opendatahub/test-curl-8013-0c983f" Sep 11 14:38:18.523236 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:18.523204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68cd4\" (UniqueName: \"kubernetes.io/projected/a1d308d4-c276-4a36-9ace-de849bc7c6d2-kube-api-access-68cd4\") pod \"test-curl-8013-0c983f\" (UID: \"a1d308d4-c276-4a36-9ace-de849bc7c6d2\") " pod="opendatahub/test-curl-8013-0c983f" Sep 11 14:38:18.534508 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:18.534476 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68cd4\" (UniqueName: \"kubernetes.io/projected/a1d308d4-c276-4a36-9ace-de849bc7c6d2-kube-api-access-68cd4\") pod \"test-curl-8013-0c983f\" (UID: \"a1d308d4-c276-4a36-9ace-de849bc7c6d2\") " pod="opendatahub/test-curl-8013-0c983f" Sep 11 14:38:18.574386 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:18.574357 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-0c983f" Sep 11 14:38:18.863121 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:18.863097 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8013-0c983f"] Sep 11 14:38:18.865352 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:38:18.865321 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1d308d4_c276_4a36_9ace_de849bc7c6d2.slice/crio-bcaf651d0c3fe93df41adac903da74a3408a34c3b12b9c5f8309fd7527d6cd9e WatchSource:0}: Error finding container bcaf651d0c3fe93df41adac903da74a3408a34c3b12b9c5f8309fd7527d6cd9e: Status 404 returned error can't find the container with id bcaf651d0c3fe93df41adac903da74a3408a34c3b12b9c5f8309fd7527d6cd9e Sep 11 14:38:19.359899 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:19.359867 2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd6d5873_4af1_4e84_b9b7_b3a4998543e7.slice/crio-conmon-ff390c21ccbcb5d1084d1aafd54ba510b6dda02029069c93dbe115f0502180ff.scope\": RecentStats: unable to find data in memory cache]" Sep 11 14:38:19.363801 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:19.363769 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8013-0c983f" event={"ID":"a1d308d4-c276-4a36-9ace-de849bc7c6d2","Type":"ContainerStarted","Data":"771802d893476ac5579fe80360f938aa9696d89a6d3d889a62609091a4bfffa2"} Sep 11 14:38:19.363923 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:19.363815 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8013-0c983f" event={"ID":"a1d308d4-c276-4a36-9ace-de849bc7c6d2","Type":"ContainerStarted","Data":"bcaf651d0c3fe93df41adac903da74a3408a34c3b12b9c5f8309fd7527d6cd9e"} Sep 11 14:38:19.364002 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:19.363951 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8013-5140b5" podUID="7b573e1d-fe19-4771-978e-650b51180f9a" containerName="test-curl-8013-5140b5" containerID="cri-o://d8c5863c38eaea437bf0c37f4ec0033fb6c616691d07130329b0685147ee64bd" gracePeriod=2 Sep 11 14:38:19.404719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:19.404668 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8013-0c983f" podStartSLOduration=1.404649398 podStartE2EDuration="1.404649398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:19.401837799 +0000 UTC m=+1387.487294158" watchObservedRunningTime="2026-09-11 14:38:19.404649398 +0000 UTC m=+1387.490105757" Sep 11 14:38:19.518935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:19.518911 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-0fedfe" Sep 11 14:38:19.640038 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:19.639944 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf77f\" (UniqueName: \"kubernetes.io/projected/cd6d5873-4af1-4e84-b9b7-b3a4998543e7-kube-api-access-rf77f\") pod \"cd6d5873-4af1-4e84-b9b7-b3a4998543e7\" (UID: \"cd6d5873-4af1-4e84-b9b7-b3a4998543e7\") " Sep 11 14:38:19.642111 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:19.642082 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6d5873-4af1-4e84-b9b7-b3a4998543e7-kube-api-access-rf77f" (OuterVolumeSpecName: "kube-api-access-rf77f") pod "cd6d5873-4af1-4e84-b9b7-b3a4998543e7" (UID: "cd6d5873-4af1-4e84-b9b7-b3a4998543e7"). InnerVolumeSpecName "kube-api-access-rf77f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:38:19.741125 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:19.741092 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf77f\" (UniqueName: \"kubernetes.io/projected/cd6d5873-4af1-4e84-b9b7-b3a4998543e7-kube-api-access-rf77f\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:38:20.079954 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.079870 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8013-0c983f"] Sep 11 14:38:20.097336 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.097307 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8013-0c983f"] Sep 11 14:38:20.228789 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.228755 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8013-56ea82"] Sep 11 14:38:20.229225 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.229211 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d308d4-c276-4a36-9ace-de849bc7c6d2" containerName="test-curl-8013-0c983f" Sep 11 14:38:20.229271 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.229227 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d308d4-c276-4a36-9ace-de849bc7c6d2" containerName="test-curl-8013-0c983f" Sep 11 14:38:20.229340 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.229329 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1d308d4-c276-4a36-9ace-de849bc7c6d2" containerName="test-curl-8013-0c983f" Sep 11 14:38:20.231398 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.231381 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-56ea82" Sep 11 14:38:20.251327 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.251298 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8013-56ea82"] Sep 11 14:38:20.346607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.346513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57rtf\" (UniqueName: \"kubernetes.io/projected/a426c96a-ee52-4cc1-810e-1855f60c1fb2-kube-api-access-57rtf\") pod \"test-curl-8013-56ea82\" (UID: \"a426c96a-ee52-4cc1-810e-1855f60c1fb2\") " pod="opendatahub/test-curl-8013-56ea82" Sep 11 14:38:20.371443 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.371405 2578 generic.go:356] "Generic (PLEG): container finished" podID="cd6d5873-4af1-4e84-b9b7-b3a4998543e7" containerID="ff390c21ccbcb5d1084d1aafd54ba510b6dda02029069c93dbe115f0502180ff" exitCode=137 Sep 11 14:38:20.371837 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.371453 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-0fedfe" Sep 11 14:38:20.371837 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.371468 2578 scope.go:117] "RemoveContainer" containerID="ff390c21ccbcb5d1084d1aafd54ba510b6dda02029069c93dbe115f0502180ff" Sep 11 14:38:20.371941 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:20.371855 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8013-0c983f" container="test-curl-8013-0c983f" Sep 11 14:38:20.381048 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.381030 2578 scope.go:117] "RemoveContainer" containerID="ff390c21ccbcb5d1084d1aafd54ba510b6dda02029069c93dbe115f0502180ff" Sep 11 14:38:20.381329 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:20.381309 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff390c21ccbcb5d1084d1aafd54ba510b6dda02029069c93dbe115f0502180ff\": container with ID starting with ff390c21ccbcb5d1084d1aafd54ba510b6dda02029069c93dbe115f0502180ff not found: ID does not exist" containerID="ff390c21ccbcb5d1084d1aafd54ba510b6dda02029069c93dbe115f0502180ff" Sep 11 14:38:20.381373 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.381339 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff390c21ccbcb5d1084d1aafd54ba510b6dda02029069c93dbe115f0502180ff"} err="failed to get container status \"ff390c21ccbcb5d1084d1aafd54ba510b6dda02029069c93dbe115f0502180ff\": rpc error: code = NotFound desc = could not find container \"ff390c21ccbcb5d1084d1aafd54ba510b6dda02029069c93dbe115f0502180ff\": container with ID starting with ff390c21ccbcb5d1084d1aafd54ba510b6dda02029069c93dbe115f0502180ff not found: ID does not exist" Sep 11 14:38:20.382049 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:20.382028 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8013-0fedfe\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="cd6d5873-4af1-4e84-b9b7-b3a4998543e7" pod="opendatahub/test-curl-8013-0fedfe" Sep 11 14:38:20.447881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.447848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57rtf\" (UniqueName: \"kubernetes.io/projected/a426c96a-ee52-4cc1-810e-1855f60c1fb2-kube-api-access-57rtf\") pod \"test-curl-8013-56ea82\" (UID: \"a426c96a-ee52-4cc1-810e-1855f60c1fb2\") " pod="opendatahub/test-curl-8013-56ea82" Sep 11 14:38:20.464220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.464185 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57rtf\" (UniqueName: \"kubernetes.io/projected/a426c96a-ee52-4cc1-810e-1855f60c1fb2-kube-api-access-57rtf\") pod \"test-curl-8013-56ea82\" (UID: \"a426c96a-ee52-4cc1-810e-1855f60c1fb2\") " pod="opendatahub/test-curl-8013-56ea82" Sep 11 14:38:20.465013 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.464987 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd6d5873-4af1-4e84-b9b7-b3a4998543e7" path="/var/lib/kubelet/pods/cd6d5873-4af1-4e84-b9b7-b3a4998543e7/volumes" Sep 11 14:38:20.540139 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.540106 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-56ea82" Sep 11 14:38:20.833712 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:20.833684 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8013-56ea82"] Sep 11 14:38:20.834752 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:38:20.834706 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda426c96a_ee52_4cc1_810e_1855f60c1fb2.slice/crio-a0d5bd192be193e880093579a430f227081092edfb2658e7785efa8632af42d2 WatchSource:0}: Error finding container a0d5bd192be193e880093579a430f227081092edfb2658e7785efa8632af42d2: Status 404 returned error can't find the container with id a0d5bd192be193e880093579a430f227081092edfb2658e7785efa8632af42d2 Sep 11 14:38:21.380569 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:21.380539 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8013-56ea82" event={"ID":"a426c96a-ee52-4cc1-810e-1855f60c1fb2","Type":"ContainerStarted","Data":"fea27af661ae9072a9cbe2f03e36cd45d3adda4bd91cdf61db1ebbe8d758f907"} Sep 11 14:38:21.381008 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:21.380585 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8013-56ea82" event={"ID":"a426c96a-ee52-4cc1-810e-1855f60c1fb2","Type":"ContainerStarted","Data":"a0d5bd192be193e880093579a430f227081092edfb2658e7785efa8632af42d2"} Sep 11 14:38:21.381721 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:21.381677 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8013-0c983f" podUID="a1d308d4-c276-4a36-9ace-de849bc7c6d2" containerName="test-curl-8013-0c983f" containerID="cri-o://771802d893476ac5579fe80360f938aa9696d89a6d3d889a62609091a4bfffa2" gracePeriod=2 Sep 11 14:38:21.413095 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:21.413046 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8013-56ea82" podStartSLOduration=1.4130304360000001 podStartE2EDuration="1.413030436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:21.410864614 +0000 UTC m=+1389.496320978" watchObservedRunningTime="2026-09-11 14:38:21.413030436 +0000 UTC m=+1389.498486840" Sep 11 14:38:21.495562 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:21.495539 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-5140b5" Sep 11 14:38:21.660269 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:21.660198 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9jrz\" (UniqueName: \"kubernetes.io/projected/7b573e1d-fe19-4771-978e-650b51180f9a-kube-api-access-v9jrz\") pod \"7b573e1d-fe19-4771-978e-650b51180f9a\" (UID: \"7b573e1d-fe19-4771-978e-650b51180f9a\") " Sep 11 14:38:21.662183 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:21.662150 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b573e1d-fe19-4771-978e-650b51180f9a-kube-api-access-v9jrz" (OuterVolumeSpecName: "kube-api-access-v9jrz") pod "7b573e1d-fe19-4771-978e-650b51180f9a" (UID: "7b573e1d-fe19-4771-978e-650b51180f9a"). InnerVolumeSpecName "kube-api-access-v9jrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:38:21.761536 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:21.761496 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9jrz\" (UniqueName: \"kubernetes.io/projected/7b573e1d-fe19-4771-978e-650b51180f9a-kube-api-access-v9jrz\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:38:22.139069 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:22.139032 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8013-56ea82"] Sep 11 14:38:22.152007 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:22.151895 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8013-56ea82"] Sep 11 14:38:22.389284 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:22.389200 2578 generic.go:356] "Generic (PLEG): container finished" podID="7b573e1d-fe19-4771-978e-650b51180f9a" containerID="d8c5863c38eaea437bf0c37f4ec0033fb6c616691d07130329b0685147ee64bd" exitCode=137 Sep 11 14:38:22.389284 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:22.389245 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-5140b5" Sep 11 14:38:22.389798 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:22.389295 2578 scope.go:117] "RemoveContainer" containerID="d8c5863c38eaea437bf0c37f4ec0033fb6c616691d07130329b0685147ee64bd" Sep 11 14:38:22.389798 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:22.389642 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8013-56ea82" container="test-curl-8013-56ea82" Sep 11 14:38:22.399106 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:22.399086 2578 scope.go:117] "RemoveContainer" containerID="d8c5863c38eaea437bf0c37f4ec0033fb6c616691d07130329b0685147ee64bd" Sep 11 14:38:22.399389 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:22.399368 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c5863c38eaea437bf0c37f4ec0033fb6c616691d07130329b0685147ee64bd\": container with ID starting with d8c5863c38eaea437bf0c37f4ec0033fb6c616691d07130329b0685147ee64bd not found: ID does not exist" containerID="d8c5863c38eaea437bf0c37f4ec0033fb6c616691d07130329b0685147ee64bd" Sep 11 14:38:22.399471 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:22.399395 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c5863c38eaea437bf0c37f4ec0033fb6c616691d07130329b0685147ee64bd"} err="failed to get container status \"d8c5863c38eaea437bf0c37f4ec0033fb6c616691d07130329b0685147ee64bd\": rpc error: code = NotFound desc = could not find container \"d8c5863c38eaea437bf0c37f4ec0033fb6c616691d07130329b0685147ee64bd\": container with ID starting with d8c5863c38eaea437bf0c37f4ec0033fb6c616691d07130329b0685147ee64bd not found: ID does not exist" Sep 11 14:38:22.400323 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:22.400293 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8013-5140b5\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="7b573e1d-fe19-4771-978e-650b51180f9a" pod="opendatahub/test-curl-8013-5140b5" Sep 11 14:38:22.463896 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:22.463843 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8013-5140b5\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="7b573e1d-fe19-4771-978e-650b51180f9a" pod="opendatahub/test-curl-8013-5140b5" Sep 11 14:38:22.465287 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:22.465261 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b573e1d-fe19-4771-978e-650b51180f9a" path="/var/lib/kubelet/pods/7b573e1d-fe19-4771-978e-650b51180f9a/volumes" Sep 11 14:38:23.235487 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.235457 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2"] Sep 11 14:38:23.235958 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.235944 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a426c96a-ee52-4cc1-810e-1855f60c1fb2" containerName="test-curl-8013-56ea82" Sep 11 14:38:23.236005 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.235960 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a426c96a-ee52-4cc1-810e-1855f60c1fb2" containerName="test-curl-8013-56ea82" Sep 11 14:38:23.236057 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.236044 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a426c96a-ee52-4cc1-810e-1855f60c1fb2" containerName="test-curl-8013-56ea82" Sep 11 14:38:23.238764 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.238749 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.242701 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.242680 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2b947f70-openshift-default-dockercfg-lgtkm\"" Sep 11 14:38:23.263306 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.263275 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2"] Sep 11 14:38:23.378355 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.378317 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-workload-socket\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.378531 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.378366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-workload-certs\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.378531 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.378381 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-token\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.378531 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.378481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-podinfo\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.378531 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.378514 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-data\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.378659 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.378537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-credential-socket\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.378659 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.378583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-envoy\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.378659 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.378604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792c9442-901a-430c-9aed-c3ad204b3ac0-istiod-ca-cert\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.378659 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.378634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgqdj\" (UniqueName: \"kubernetes.io/projected/792c9442-901a-430c-9aed-c3ad204b3ac0-kube-api-access-pgqdj\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.398176 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.398145 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8013-56ea82" podUID="a426c96a-ee52-4cc1-810e-1855f60c1fb2" containerName="test-curl-8013-56ea82" containerID="cri-o://fea27af661ae9072a9cbe2f03e36cd45d3adda4bd91cdf61db1ebbe8d758f907" gracePeriod=2 Sep 11 14:38:23.479877 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.479834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-credential-socket\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.479877 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.479881 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-envoy\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.480082 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.479903 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792c9442-901a-430c-9aed-c3ad204b3ac0-istiod-ca-cert\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.480082 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.479927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgqdj\" (UniqueName: \"kubernetes.io/projected/792c9442-901a-430c-9aed-c3ad204b3ac0-kube-api-access-pgqdj\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.480082 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.479970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-workload-socket\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.480292 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.480264 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-credential-socket\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.480397 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.480292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-workload-certs\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.480397 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.480343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-token\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.480504 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.480416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-podinfo\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.480504 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.480347 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-workload-socket\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.480504 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.480488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-data\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.480661 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.480572 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-workload-certs\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.480999 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.480979 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-data\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.481168 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.481149 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792c9442-901a-430c-9aed-c3ad204b3ac0-istiod-ca-cert\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.482610 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.482566 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-envoy\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.482891 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.482874 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-podinfo\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.492540 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.492439 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-token\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.492540 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.492494 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgqdj\" (UniqueName: \"kubernetes.io/projected/792c9442-901a-430c-9aed-c3ad204b3ac0-kube-api-access-pgqdj\") pod \"e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.548871 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.548842 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:23.580032 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.580003 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-0c983f" Sep 11 14:38:23.682215 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.682186 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68cd4\" (UniqueName: \"kubernetes.io/projected/a1d308d4-c276-4a36-9ace-de849bc7c6d2-kube-api-access-68cd4\") pod \"a1d308d4-c276-4a36-9ace-de849bc7c6d2\" (UID: \"a1d308d4-c276-4a36-9ace-de849bc7c6d2\") " Sep 11 14:38:23.684264 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.684238 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d308d4-c276-4a36-9ace-de849bc7c6d2-kube-api-access-68cd4" (OuterVolumeSpecName: "kube-api-access-68cd4") pod "a1d308d4-c276-4a36-9ace-de849bc7c6d2" (UID: "a1d308d4-c276-4a36-9ace-de849bc7c6d2"). InnerVolumeSpecName "kube-api-access-68cd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:38:23.783281 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.783199 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68cd4\" (UniqueName: \"kubernetes.io/projected/a1d308d4-c276-4a36-9ace-de849bc7c6d2-kube-api-access-68cd4\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:38:23.835574 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.835549 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2"] Sep 11 14:38:23.836777 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:38:23.836710 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792c9442_901a_430c_9aed_c3ad204b3ac0.slice/crio-acf087fe3b05a6b96747bd5aeea75fd3a07136e114c5dec27a4f8880f4570b8e WatchSource:0}: Error finding container acf087fe3b05a6b96747bd5aeea75fd3a07136e114c5dec27a4f8880f4570b8e: Status 404 returned error can't find the container with id acf087fe3b05a6b96747bd5aeea75fd3a07136e114c5dec27a4f8880f4570b8e Sep 11 14:38:23.838986 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.838953 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:23.839062 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.839018 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:23.839062 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:23.839045 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:24.406051 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:24.406012 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" event={"ID":"792c9442-901a-430c-9aed-c3ad204b3ac0","Type":"ContainerStarted","Data":"f3447afba9a5c6eec21f2583f91409323e9612db585f31e9534fc150e56ef7c7"} Sep 11 14:38:24.406470 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:24.406075 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" event={"ID":"792c9442-901a-430c-9aed-c3ad204b3ac0","Type":"ContainerStarted","Data":"acf087fe3b05a6b96747bd5aeea75fd3a07136e114c5dec27a4f8880f4570b8e"} Sep 11 14:38:24.407441 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:24.407409 2578 generic.go:356] "Generic (PLEG): container finished" podID="a1d308d4-c276-4a36-9ace-de849bc7c6d2" containerID="771802d893476ac5579fe80360f938aa9696d89a6d3d889a62609091a4bfffa2" exitCode=137 Sep 11 14:38:24.407549 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:24.407444 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-0c983f" Sep 11 14:38:24.407549 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:24.407470 2578 scope.go:117] "RemoveContainer" containerID="771802d893476ac5579fe80360f938aa9696d89a6d3d889a62609091a4bfffa2" Sep 11 14:38:24.417565 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:24.417543 2578 scope.go:117] "RemoveContainer" containerID="771802d893476ac5579fe80360f938aa9696d89a6d3d889a62609091a4bfffa2" Sep 11 14:38:24.417870 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:24.417842 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771802d893476ac5579fe80360f938aa9696d89a6d3d889a62609091a4bfffa2\": container with ID starting with 771802d893476ac5579fe80360f938aa9696d89a6d3d889a62609091a4bfffa2 not found: ID does not exist" containerID="771802d893476ac5579fe80360f938aa9696d89a6d3d889a62609091a4bfffa2" Sep 11 14:38:24.417935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:24.417870 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771802d893476ac5579fe80360f938aa9696d89a6d3d889a62609091a4bfffa2"} err="failed to get container status \"771802d893476ac5579fe80360f938aa9696d89a6d3d889a62609091a4bfffa2\": rpc error: code = NotFound desc = could not find container \"771802d893476ac5579fe80360f938aa9696d89a6d3d889a62609091a4bfffa2\": container with ID starting with 771802d893476ac5579fe80360f938aa9696d89a6d3d889a62609091a4bfffa2 not found: ID does not exist" Sep 11 14:38:24.440307 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:24.440263 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" podStartSLOduration=1.440248955 podStartE2EDuration="1.440248955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:24.438707035 +0000 UTC m=+1392.524163394" watchObservedRunningTime="2026-09-11 14:38:24.440248955 +0000 UTC m=+1392.525705315" Sep 11 14:38:24.441108 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:24.441077 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8013-0c983f\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="a1d308d4-c276-4a36-9ace-de849bc7c6d2" pod="opendatahub/test-curl-8013-0c983f" Sep 11 14:38:24.465156 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:24.465128 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1d308d4-c276-4a36-9ace-de849bc7c6d2" path="/var/lib/kubelet/pods/a1d308d4-c276-4a36-9ace-de849bc7c6d2/volumes" Sep 11 14:38:24.549071 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:24.549030 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:24.555015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:24.554985 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:25.415317 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:25.415287 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:25.416225 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:25.416208 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:25.598363 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:25.598335 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-56ea82" Sep 11 14:38:25.702865 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:25.702779 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57rtf\" (UniqueName: \"kubernetes.io/projected/a426c96a-ee52-4cc1-810e-1855f60c1fb2-kube-api-access-57rtf\") pod \"a426c96a-ee52-4cc1-810e-1855f60c1fb2\" (UID: \"a426c96a-ee52-4cc1-810e-1855f60c1fb2\") " Sep 11 14:38:25.704887 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:25.704860 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a426c96a-ee52-4cc1-810e-1855f60c1fb2-kube-api-access-57rtf" (OuterVolumeSpecName: "kube-api-access-57rtf") pod "a426c96a-ee52-4cc1-810e-1855f60c1fb2" (UID: "a426c96a-ee52-4cc1-810e-1855f60c1fb2"). InnerVolumeSpecName "kube-api-access-57rtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:38:25.803820 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:25.803785 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57rtf\" (UniqueName: \"kubernetes.io/projected/a426c96a-ee52-4cc1-810e-1855f60c1fb2-kube-api-access-57rtf\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:38:26.422757 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:26.422700 2578 generic.go:356] "Generic (PLEG): container finished" podID="a426c96a-ee52-4cc1-810e-1855f60c1fb2" containerID="fea27af661ae9072a9cbe2f03e36cd45d3adda4bd91cdf61db1ebbe8d758f907" exitCode=137 Sep 11 14:38:26.422757 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:26.422754 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8013-56ea82" Sep 11 14:38:26.423249 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:26.422772 2578 scope.go:117] "RemoveContainer" containerID="fea27af661ae9072a9cbe2f03e36cd45d3adda4bd91cdf61db1ebbe8d758f907" Sep 11 14:38:26.432286 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:26.432268 2578 scope.go:117] "RemoveContainer" containerID="fea27af661ae9072a9cbe2f03e36cd45d3adda4bd91cdf61db1ebbe8d758f907" Sep 11 14:38:26.432583 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:26.432563 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea27af661ae9072a9cbe2f03e36cd45d3adda4bd91cdf61db1ebbe8d758f907\": container with ID starting with fea27af661ae9072a9cbe2f03e36cd45d3adda4bd91cdf61db1ebbe8d758f907 not found: ID does not exist" containerID="fea27af661ae9072a9cbe2f03e36cd45d3adda4bd91cdf61db1ebbe8d758f907" Sep 11 14:38:26.432635 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:26.432594 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea27af661ae9072a9cbe2f03e36cd45d3adda4bd91cdf61db1ebbe8d758f907"} err="failed to get container status \"fea27af661ae9072a9cbe2f03e36cd45d3adda4bd91cdf61db1ebbe8d758f907\": rpc error: code = NotFound desc = could not find container \"fea27af661ae9072a9cbe2f03e36cd45d3adda4bd91cdf61db1ebbe8d758f907\": container with ID starting with fea27af661ae9072a9cbe2f03e36cd45d3adda4bd91cdf61db1ebbe8d758f907 not found: ID does not exist" Sep 11 14:38:26.434066 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:38:26.434046 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8013-56ea82\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="a426c96a-ee52-4cc1-810e-1855f60c1fb2" pod="opendatahub/test-curl-8013-56ea82" Sep 11 14:38:26.464719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:26.464692 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a426c96a-ee52-4cc1-810e-1855f60c1fb2" path="/var/lib/kubelet/pods/a426c96a-ee52-4cc1-810e-1855f60c1fb2/volumes" Sep 11 14:38:28.244993 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:28.244945 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:38:44.752724 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:44.752689 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:38:55.416765 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:55.416707 2578 patch_prober.go:28] interesting pod/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 11 14:38:55.417176 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:55.416814 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 11 14:38:55.633093 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:55.633064 2578 generic.go:356] "Generic (PLEG): container finished" podID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerID="f3447afba9a5c6eec21f2583f91409323e9612db585f31e9534fc150e56ef7c7" exitCode=0 Sep 11 14:38:55.633257 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:55.633141 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" event={"ID":"792c9442-901a-430c-9aed-c3ad204b3ac0","Type":"ContainerDied","Data":"f3447afba9a5c6eec21f2583f91409323e9612db585f31e9534fc150e56ef7c7"} Sep 11 14:38:55.633588 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:55.633573 2578 scope.go:117] "RemoveContainer" containerID="f3447afba9a5c6eec21f2583f91409323e9612db585f31e9534fc150e56ef7c7" Sep 11 14:38:55.634709 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:55.634678 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:55.634835 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:55.634775 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:55.634835 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:55.634811 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:56.641890 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:56.641857 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" event={"ID":"792c9442-901a-430c-9aed-c3ad204b3ac0","Type":"ContainerStarted","Data":"1f1f19a9d11c9cbdfce507644db3bb0121ed4c967a7c7b87274be16065a69714"} Sep 11 14:38:57.550171 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:57.550136 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:38:57.550545 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:57.550453 2578 patch_prober.go:28] interesting pod/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 11 14:38:57.551183 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:57.550690 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 11 14:38:58.549662 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:58.549619 2578 patch_prober.go:28] interesting pod/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 11 14:38:58.550169 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:58.549690 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 11 14:38:59.558870 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:59.558825 2578 patch_prober.go:28] interesting pod/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 11 14:38:59.559339 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:38:59.558894 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 11 14:39:00.549685 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:00.549650 2578 patch_prober.go:28] interesting pod/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 11 14:39:00.550002 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:00.549972 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 11 14:39:01.549566 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:01.549528 2578 patch_prober.go:28] interesting pod/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 11 14:39:01.550203 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:01.550155 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 11 14:39:02.553762 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:02.553711 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:39:02.554158 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:02.553990 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:39:02.554763 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:02.554724 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:39:04.248686 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:04.248653 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:39:12.536935 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:12.536858 2578 scope.go:117] "RemoveContainer" containerID="81de7670a54c5ffdead0f72fa86d370ebae45b3c392f897742e1b15122b62493" Sep 11 14:39:21.859792 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:21.859748 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:39:47.251339 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:47.251306 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:39:59.536677 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:59.536645 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8016-00ff64"] Sep 11 14:39:59.540445 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:59.540428 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8016-00ff64" Sep 11 14:39:59.544124 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:59.544104 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7pn49\"" Sep 11 14:39:59.552138 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:59.552115 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8016-00ff64"] Sep 11 14:39:59.595661 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:59.595631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnm7n\" (UniqueName: \"kubernetes.io/projected/4f906cd5-8845-4918-8c20-b7ea490594e4-kube-api-access-wnm7n\") pod \"test-curl-8016-00ff64\" (UID: \"4f906cd5-8845-4918-8c20-b7ea490594e4\") " pod="opendatahub/test-curl-8016-00ff64" Sep 11 14:39:59.696931 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:59.696899 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnm7n\" (UniqueName: \"kubernetes.io/projected/4f906cd5-8845-4918-8c20-b7ea490594e4-kube-api-access-wnm7n\") pod \"test-curl-8016-00ff64\" (UID: \"4f906cd5-8845-4918-8c20-b7ea490594e4\") " pod="opendatahub/test-curl-8016-00ff64" Sep 11 14:39:59.709619 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:59.709592 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnm7n\" (UniqueName: \"kubernetes.io/projected/4f906cd5-8845-4918-8c20-b7ea490594e4-kube-api-access-wnm7n\") pod \"test-curl-8016-00ff64\" (UID: \"4f906cd5-8845-4918-8c20-b7ea490594e4\") " pod="opendatahub/test-curl-8016-00ff64" Sep 11 14:39:59.849568 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:39:59.849530 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8016-00ff64" Sep 11 14:40:00.132061 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:00.132037 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8016-00ff64"] Sep 11 14:40:00.133798 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:40:00.133768 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f906cd5_8845_4918_8c20_b7ea490594e4.slice/crio-7659eabed239254f773762c3ad0682affb60b9b512b4ca81eecc2652cfc4c038 WatchSource:0}: Error finding container 7659eabed239254f773762c3ad0682affb60b9b512b4ca81eecc2652cfc4c038: Status 404 returned error can't find the container with id 7659eabed239254f773762c3ad0682affb60b9b512b4ca81eecc2652cfc4c038 Sep 11 14:40:01.048212 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:01.048177 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8016-00ff64" event={"ID":"4f906cd5-8845-4918-8c20-b7ea490594e4","Type":"ContainerStarted","Data":"241ddd9479f933baab752a0f38ec5d8d897cc8dfb3f3b504e20b7464cf556e82"} Sep 11 14:40:01.048576 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:01.048220 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8016-00ff64" event={"ID":"4f906cd5-8845-4918-8c20-b7ea490594e4","Type":"ContainerStarted","Data":"7659eabed239254f773762c3ad0682affb60b9b512b4ca81eecc2652cfc4c038"} Sep 11 14:40:01.074961 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:01.074911 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8016-00ff64" podStartSLOduration=2.074896115 podStartE2EDuration="2.074896115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:01.074232252 +0000 UTC m=+1489.159688613" watchObservedRunningTime="2026-09-11 14:40:01.074896115 +0000 UTC m=+1489.160352475" Sep 11 14:40:01.796685 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:01.796649 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8016-00ff64"] Sep 11 14:40:01.807026 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:01.806952 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8016-00ff64"] Sep 11 14:40:02.055986 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:40:02.055957 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8016-00ff64" container="test-curl-8016-00ff64" Sep 11 14:40:03.063109 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:03.063068 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8016-00ff64" podUID="4f906cd5-8845-4918-8c20-b7ea490594e4" containerName="test-curl-8016-00ff64" containerID="cri-o://241ddd9479f933baab752a0f38ec5d8d897cc8dfb3f3b504e20b7464cf556e82" gracePeriod=2 Sep 11 14:40:04.372022 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:04.371982 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:40:05.249244 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:05.249208 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8016-00ff64" Sep 11 14:40:05.350688 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:05.350654 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnm7n\" (UniqueName: \"kubernetes.io/projected/4f906cd5-8845-4918-8c20-b7ea490594e4-kube-api-access-wnm7n\") pod \"4f906cd5-8845-4918-8c20-b7ea490594e4\" (UID: \"4f906cd5-8845-4918-8c20-b7ea490594e4\") " Sep 11 14:40:05.352854 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:05.352822 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f906cd5-8845-4918-8c20-b7ea490594e4-kube-api-access-wnm7n" (OuterVolumeSpecName: "kube-api-access-wnm7n") pod "4f906cd5-8845-4918-8c20-b7ea490594e4" (UID: "4f906cd5-8845-4918-8c20-b7ea490594e4"). InnerVolumeSpecName "kube-api-access-wnm7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:40:05.452314 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:05.452283 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnm7n\" (UniqueName: \"kubernetes.io/projected/4f906cd5-8845-4918-8c20-b7ea490594e4-kube-api-access-wnm7n\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:40:06.083158 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:06.083127 2578 generic.go:356] "Generic (PLEG): container finished" podID="4f906cd5-8845-4918-8c20-b7ea490594e4" containerID="241ddd9479f933baab752a0f38ec5d8d897cc8dfb3f3b504e20b7464cf556e82" exitCode=137 Sep 11 14:40:06.083312 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:06.083167 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8016-00ff64" Sep 11 14:40:06.083312 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:06.083227 2578 scope.go:117] "RemoveContainer" containerID="241ddd9479f933baab752a0f38ec5d8d897cc8dfb3f3b504e20b7464cf556e82" Sep 11 14:40:06.093504 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:06.093485 2578 scope.go:117] "RemoveContainer" containerID="241ddd9479f933baab752a0f38ec5d8d897cc8dfb3f3b504e20b7464cf556e82" Sep 11 14:40:06.093817 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:40:06.093798 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"241ddd9479f933baab752a0f38ec5d8d897cc8dfb3f3b504e20b7464cf556e82\": container with ID starting with 241ddd9479f933baab752a0f38ec5d8d897cc8dfb3f3b504e20b7464cf556e82 not found: ID does not exist" containerID="241ddd9479f933baab752a0f38ec5d8d897cc8dfb3f3b504e20b7464cf556e82" Sep 11 14:40:06.093910 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:06.093823 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241ddd9479f933baab752a0f38ec5d8d897cc8dfb3f3b504e20b7464cf556e82"} err="failed to get container status \"241ddd9479f933baab752a0f38ec5d8d897cc8dfb3f3b504e20b7464cf556e82\": rpc error: code = NotFound desc = could not find container \"241ddd9479f933baab752a0f38ec5d8d897cc8dfb3f3b504e20b7464cf556e82\": container with ID starting with 241ddd9479f933baab752a0f38ec5d8d897cc8dfb3f3b504e20b7464cf556e82 not found: ID does not exist" Sep 11 14:40:06.095315 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:40:06.095292 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8016-00ff64\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="4f906cd5-8845-4918-8c20-b7ea490594e4" pod="opendatahub/test-curl-8016-00ff64" Sep 11 14:40:06.465912 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:06.465881 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f906cd5-8845-4918-8c20-b7ea490594e4" path="/var/lib/kubelet/pods/4f906cd5-8845-4918-8c20-b7ea490594e4/volumes" Sep 11 14:40:12.414831 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:12.414802 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:40:12.415315 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:12.415298 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:40:16.932909 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:16.932873 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:40:32.349660 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:32.349621 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:40:49.659291 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:40:49.659206 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:41:04.074015 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:04.073981 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:41:05.084174 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:05.084123 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2"] Sep 11 14:41:05.084638 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:05.084390 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerName="istio-proxy" containerID="cri-o://1f1f19a9d11c9cbdfce507644db3bb0121ed4c967a7c7b87274be16065a69714" gracePeriod=30 Sep 11 14:41:10.341997 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.341972 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:41:10.457954 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.457915 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-workload-socket\") pod \"792c9442-901a-430c-9aed-c3ad204b3ac0\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " Sep 11 14:41:10.458157 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.457988 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-token\") pod \"792c9442-901a-430c-9aed-c3ad204b3ac0\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " Sep 11 14:41:10.458157 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458016 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-workload-certs\") pod \"792c9442-901a-430c-9aed-c3ad204b3ac0\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " Sep 11 14:41:10.458157 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458031 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-envoy\") pod \"792c9442-901a-430c-9aed-c3ad204b3ac0\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " Sep 11 14:41:10.458157 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458046 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792c9442-901a-430c-9aed-c3ad204b3ac0-istiod-ca-cert\") pod \"792c9442-901a-430c-9aed-c3ad204b3ac0\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " Sep 11 14:41:10.458157 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458076 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgqdj\" (UniqueName: \"kubernetes.io/projected/792c9442-901a-430c-9aed-c3ad204b3ac0-kube-api-access-pgqdj\") pod \"792c9442-901a-430c-9aed-c3ad204b3ac0\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " Sep 11 14:41:10.458157 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458119 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-podinfo\") pod \"792c9442-901a-430c-9aed-c3ad204b3ac0\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " Sep 11 14:41:10.458157 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458146 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-credential-socket\") pod \"792c9442-901a-430c-9aed-c3ad204b3ac0\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " Sep 11 14:41:10.458548 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458175 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-data\") pod \"792c9442-901a-430c-9aed-c3ad204b3ac0\" (UID: \"792c9442-901a-430c-9aed-c3ad204b3ac0\") " Sep 11 14:41:10.458548 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458184 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "792c9442-901a-430c-9aed-c3ad204b3ac0" (UID: "792c9442-901a-430c-9aed-c3ad204b3ac0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:41:10.458548 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458315 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "792c9442-901a-430c-9aed-c3ad204b3ac0" (UID: "792c9442-901a-430c-9aed-c3ad204b3ac0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:41:10.458548 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458533 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-workload-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:41:10.458790 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458550 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-workload-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:41:10.458790 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458614 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "792c9442-901a-430c-9aed-c3ad204b3ac0" (UID: "792c9442-901a-430c-9aed-c3ad204b3ac0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:41:10.458790 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458758 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-data" (OuterVolumeSpecName: "istio-data") pod "792c9442-901a-430c-9aed-c3ad204b3ac0" (UID: "792c9442-901a-430c-9aed-c3ad204b3ac0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:41:10.458962 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.458859 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792c9442-901a-430c-9aed-c3ad204b3ac0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "792c9442-901a-430c-9aed-c3ad204b3ac0" (UID: "792c9442-901a-430c-9aed-c3ad204b3ac0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:41:10.460379 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.460354 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "792c9442-901a-430c-9aed-c3ad204b3ac0" (UID: "792c9442-901a-430c-9aed-c3ad204b3ac0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:41:10.460672 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.460639 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792c9442-901a-430c-9aed-c3ad204b3ac0-kube-api-access-pgqdj" (OuterVolumeSpecName: "kube-api-access-pgqdj") pod "792c9442-901a-430c-9aed-c3ad204b3ac0" (UID: "792c9442-901a-430c-9aed-c3ad204b3ac0"). InnerVolumeSpecName "kube-api-access-pgqdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:41:10.460809 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.460705 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-token" (OuterVolumeSpecName: "istio-token") pod "792c9442-901a-430c-9aed-c3ad204b3ac0" (UID: "792c9442-901a-430c-9aed-c3ad204b3ac0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:41:10.460957 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.460938 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "792c9442-901a-430c-9aed-c3ad204b3ac0" (UID: "792c9442-901a-430c-9aed-c3ad204b3ac0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:41:10.485995 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.485969 2578 generic.go:356] "Generic (PLEG): container finished" podID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerID="1f1f19a9d11c9cbdfce507644db3bb0121ed4c967a7c7b87274be16065a69714" exitCode=0 Sep 11 14:41:10.486094 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.486016 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" Sep 11 14:41:10.486094 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.486053 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" event={"ID":"792c9442-901a-430c-9aed-c3ad204b3ac0","Type":"ContainerDied","Data":"1f1f19a9d11c9cbdfce507644db3bb0121ed4c967a7c7b87274be16065a69714"} Sep 11 14:41:10.486168 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.486107 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2" event={"ID":"792c9442-901a-430c-9aed-c3ad204b3ac0","Type":"ContainerDied","Data":"acf087fe3b05a6b96747bd5aeea75fd3a07136e114c5dec27a4f8880f4570b8e"} Sep 11 14:41:10.486168 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.486130 2578 scope.go:117] "RemoveContainer" containerID="1f1f19a9d11c9cbdfce507644db3bb0121ed4c967a7c7b87274be16065a69714" Sep 11 14:41:10.495446 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.495429 2578 scope.go:117] "RemoveContainer" containerID="f3447afba9a5c6eec21f2583f91409323e9612db585f31e9534fc150e56ef7c7" Sep 11 14:41:10.504075 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.504057 2578 scope.go:117] "RemoveContainer" containerID="1f1f19a9d11c9cbdfce507644db3bb0121ed4c967a7c7b87274be16065a69714" Sep 11 14:41:10.504343 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:41:10.504325 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1f19a9d11c9cbdfce507644db3bb0121ed4c967a7c7b87274be16065a69714\": container with ID starting with 1f1f19a9d11c9cbdfce507644db3bb0121ed4c967a7c7b87274be16065a69714 not found: ID does not exist" containerID="1f1f19a9d11c9cbdfce507644db3bb0121ed4c967a7c7b87274be16065a69714" Sep 11 14:41:10.504394 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.504354 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1f19a9d11c9cbdfce507644db3bb0121ed4c967a7c7b87274be16065a69714"} err="failed to get container status \"1f1f19a9d11c9cbdfce507644db3bb0121ed4c967a7c7b87274be16065a69714\": rpc error: code = NotFound desc = could not find container \"1f1f19a9d11c9cbdfce507644db3bb0121ed4c967a7c7b87274be16065a69714\": container with ID starting with 1f1f19a9d11c9cbdfce507644db3bb0121ed4c967a7c7b87274be16065a69714 not found: ID does not exist" Sep 11 14:41:10.504394 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.504372 2578 scope.go:117] "RemoveContainer" containerID="f3447afba9a5c6eec21f2583f91409323e9612db585f31e9534fc150e56ef7c7" Sep 11 14:41:10.509748 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:41:10.505194 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3447afba9a5c6eec21f2583f91409323e9612db585f31e9534fc150e56ef7c7\": container with ID starting with f3447afba9a5c6eec21f2583f91409323e9612db585f31e9534fc150e56ef7c7 not found: ID does not exist" containerID="f3447afba9a5c6eec21f2583f91409323e9612db585f31e9534fc150e56ef7c7" Sep 11 14:41:10.509748 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.505244 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3447afba9a5c6eec21f2583f91409323e9612db585f31e9534fc150e56ef7c7"} err="failed to get container status \"f3447afba9a5c6eec21f2583f91409323e9612db585f31e9534fc150e56ef7c7\": rpc error: code = NotFound desc = could not find container \"f3447afba9a5c6eec21f2583f91409323e9612db585f31e9534fc150e56ef7c7\": container with ID starting with f3447afba9a5c6eec21f2583f91409323e9612db585f31e9534fc150e56ef7c7 not found: ID does not exist" Sep 11 14:41:10.515618 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.515595 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2"] Sep 11 14:41:10.521872 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.521851 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b947f70-openshift-default-584c4888d-b6xq2"] Sep 11 14:41:10.559560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.559536 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-envoy\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:41:10.559560 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.559556 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792c9442-901a-430c-9aed-c3ad204b3ac0-istiod-ca-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:41:10.559685 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.559565 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgqdj\" (UniqueName: \"kubernetes.io/projected/792c9442-901a-430c-9aed-c3ad204b3ac0-kube-api-access-pgqdj\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:41:10.559685 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.559575 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-podinfo\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:41:10.559685 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.559584 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-credential-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:41:10.559685 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.559592 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-data\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:41:10.559685 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:10.559600 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792c9442-901a-430c-9aed-c3ad204b3ac0-istio-token\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:41:12.466453 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:12.466412 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" path="/var/lib/kubelet/pods/792c9442-901a-430c-9aed-c3ad204b3ac0/volumes" Sep 11 14:41:18.564932 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.564895 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8016-cd8b12"] Sep 11 14:41:18.565324 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.565314 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f906cd5-8845-4918-8c20-b7ea490594e4" containerName="test-curl-8016-00ff64" Sep 11 14:41:18.565368 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.565326 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f906cd5-8845-4918-8c20-b7ea490594e4" containerName="test-curl-8016-00ff64" Sep 11 14:41:18.565368 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.565342 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerName="istio-proxy" Sep 11 14:41:18.565368 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.565348 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerName="istio-proxy" Sep 11 14:41:18.565490 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.565426 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerName="istio-proxy" Sep 11 14:41:18.565490 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.565435 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerName="istio-proxy" Sep 11 14:41:18.565490 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.565441 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f906cd5-8845-4918-8c20-b7ea490594e4" containerName="test-curl-8016-00ff64" Sep 11 14:41:18.568577 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.568558 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8016-cd8b12" Sep 11 14:41:18.572764 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.572745 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7pn49\"" Sep 11 14:41:18.587588 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.587561 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8016-cd8b12"] Sep 11 14:41:18.737115 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.737077 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgtxv\" (UniqueName: \"kubernetes.io/projected/d807d28d-b1d2-48c4-9bd4-bb403861afc7-kube-api-access-sgtxv\") pod \"test-curl-8016-cd8b12\" (UID: \"d807d28d-b1d2-48c4-9bd4-bb403861afc7\") " pod="opendatahub/test-curl-8016-cd8b12" Sep 11 14:41:18.838237 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.838150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgtxv\" (UniqueName: \"kubernetes.io/projected/d807d28d-b1d2-48c4-9bd4-bb403861afc7-kube-api-access-sgtxv\") pod \"test-curl-8016-cd8b12\" (UID: \"d807d28d-b1d2-48c4-9bd4-bb403861afc7\") " pod="opendatahub/test-curl-8016-cd8b12" Sep 11 14:41:18.849905 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.849882 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgtxv\" (UniqueName: \"kubernetes.io/projected/d807d28d-b1d2-48c4-9bd4-bb403861afc7-kube-api-access-sgtxv\") pod \"test-curl-8016-cd8b12\" (UID: \"d807d28d-b1d2-48c4-9bd4-bb403861afc7\") " pod="opendatahub/test-curl-8016-cd8b12" Sep 11 14:41:18.877010 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:18.876986 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8016-cd8b12" Sep 11 14:41:19.160313 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:19.160287 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8016-cd8b12"] Sep 11 14:41:19.162346 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:41:19.162312 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd807d28d_b1d2_48c4_9bd4_bb403861afc7.slice/crio-100763765bb927195522205b4fb4c5423c2cc25a4dc887a885e22c454c7a051d WatchSource:0}: Error finding container 100763765bb927195522205b4fb4c5423c2cc25a4dc887a885e22c454c7a051d: Status 404 returned error can't find the container with id 100763765bb927195522205b4fb4c5423c2cc25a4dc887a885e22c454c7a051d Sep 11 14:41:19.544555 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:19.544446 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8016-cd8b12" event={"ID":"d807d28d-b1d2-48c4-9bd4-bb403861afc7","Type":"ContainerStarted","Data":"e9a54c44027a19bd48c05ad638793390ae5c15cabc53cdaa0f726b07e9736888"} Sep 11 14:41:19.544555 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:19.544498 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8016-cd8b12" event={"ID":"d807d28d-b1d2-48c4-9bd4-bb403861afc7","Type":"ContainerStarted","Data":"100763765bb927195522205b4fb4c5423c2cc25a4dc887a885e22c454c7a051d"} Sep 11 14:41:19.571354 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:19.571297 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8016-cd8b12" podStartSLOduration=1.571280759 podStartE2EDuration="1.571280759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:19.568876697 +0000 UTC m=+1567.654333059" watchObservedRunningTime="2026-09-11 14:41:19.571280759 +0000 UTC m=+1567.656737118" Sep 11 14:41:20.220076 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.220040 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8016-cd8b12"] Sep 11 14:41:20.225623 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.225556 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8016-cd8b12"] Sep 11 14:41:20.355599 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.355563 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8016-b1639c"] Sep 11 14:41:20.356031 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.356016 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d807d28d-b1d2-48c4-9bd4-bb403861afc7" containerName="test-curl-8016-cd8b12" Sep 11 14:41:20.356031 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.356032 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d807d28d-b1d2-48c4-9bd4-bb403861afc7" containerName="test-curl-8016-cd8b12" Sep 11 14:41:20.356120 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.356051 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerName="istio-proxy" Sep 11 14:41:20.356120 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.356057 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="792c9442-901a-430c-9aed-c3ad204b3ac0" containerName="istio-proxy" Sep 11 14:41:20.356184 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.356123 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d807d28d-b1d2-48c4-9bd4-bb403861afc7" containerName="test-curl-8016-cd8b12" Sep 11 14:41:20.367966 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.367935 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8016-b1639c" Sep 11 14:41:20.373471 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.373442 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8016-b1639c"] Sep 11 14:41:20.454149 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.454120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9shd\" (UniqueName: \"kubernetes.io/projected/12cff6b4-1879-4541-9b8c-9403fa4ce9f5-kube-api-access-h9shd\") pod \"test-curl-8016-b1639c\" (UID: \"12cff6b4-1879-4541-9b8c-9403fa4ce9f5\") " pod="opendatahub/test-curl-8016-b1639c" Sep 11 14:41:20.551319 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:41:20.551291 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8016-cd8b12" container="test-curl-8016-cd8b12" Sep 11 14:41:20.555503 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.555483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9shd\" (UniqueName: \"kubernetes.io/projected/12cff6b4-1879-4541-9b8c-9403fa4ce9f5-kube-api-access-h9shd\") pod \"test-curl-8016-b1639c\" (UID: \"12cff6b4-1879-4541-9b8c-9403fa4ce9f5\") " pod="opendatahub/test-curl-8016-b1639c" Sep 11 14:41:20.568154 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.568121 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9shd\" (UniqueName: \"kubernetes.io/projected/12cff6b4-1879-4541-9b8c-9403fa4ce9f5-kube-api-access-h9shd\") pod \"test-curl-8016-b1639c\" (UID: \"12cff6b4-1879-4541-9b8c-9403fa4ce9f5\") " pod="opendatahub/test-curl-8016-b1639c" Sep 11 14:41:20.678978 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.678904 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8016-b1639c" Sep 11 14:41:20.960809 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:20.960783 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8016-b1639c"] Sep 11 14:41:20.963006 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:41:20.962975 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12cff6b4_1879_4541_9b8c_9403fa4ce9f5.slice/crio-4008237f2ad62002fbecf2d13f2cd9bc00cc564f5013711e161e34692e37342f WatchSource:0}: Error finding container 4008237f2ad62002fbecf2d13f2cd9bc00cc564f5013711e161e34692e37342f: Status 404 returned error can't find the container with id 4008237f2ad62002fbecf2d13f2cd9bc00cc564f5013711e161e34692e37342f Sep 11 14:41:21.559680 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:21.559642 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8016-b1639c" event={"ID":"12cff6b4-1879-4541-9b8c-9403fa4ce9f5","Type":"ContainerStarted","Data":"3a93d11189f7a52e62154d61fc72842380399eac67c917bd60b86167cfbe6084"} Sep 11 14:41:21.559680 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:21.559671 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8016-cd8b12" podUID="d807d28d-b1d2-48c4-9bd4-bb403861afc7" containerName="test-curl-8016-cd8b12" containerID="cri-o://e9a54c44027a19bd48c05ad638793390ae5c15cabc53cdaa0f726b07e9736888" gracePeriod=2 Sep 11 14:41:21.559951 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:21.559688 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8016-b1639c" event={"ID":"12cff6b4-1879-4541-9b8c-9403fa4ce9f5","Type":"ContainerStarted","Data":"4008237f2ad62002fbecf2d13f2cd9bc00cc564f5013711e161e34692e37342f"} Sep 11 14:41:21.626287 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:21.626241 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8016-b1639c" podStartSLOduration=1.626225132 podStartE2EDuration="1.626225132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:21.624204748 +0000 UTC m=+1569.709661109" watchObservedRunningTime="2026-09-11 14:41:21.626225132 +0000 UTC m=+1569.711681492" Sep 11 14:41:22.306311 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:22.306277 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8016-b1639c"] Sep 11 14:41:22.321657 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:22.321585 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8016-b1639c"] Sep 11 14:41:22.565782 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:41:22.565751 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8016-b1639c" container="test-curl-8016-b1639c" Sep 11 14:41:23.127396 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:23.127364 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:41:23.574831 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:23.574782 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8016-b1639c" podUID="12cff6b4-1879-4541-9b8c-9403fa4ce9f5" containerName="test-curl-8016-b1639c" containerID="cri-o://3a93d11189f7a52e62154d61fc72842380399eac67c917bd60b86167cfbe6084" gracePeriod=2 Sep 11 14:41:23.713157 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:23.713136 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8016-cd8b12" Sep 11 14:41:23.886111 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:23.886079 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgtxv\" (UniqueName: \"kubernetes.io/projected/d807d28d-b1d2-48c4-9bd4-bb403861afc7-kube-api-access-sgtxv\") pod \"d807d28d-b1d2-48c4-9bd4-bb403861afc7\" (UID: \"d807d28d-b1d2-48c4-9bd4-bb403861afc7\") " Sep 11 14:41:23.888089 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:23.888063 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d807d28d-b1d2-48c4-9bd4-bb403861afc7-kube-api-access-sgtxv" (OuterVolumeSpecName: "kube-api-access-sgtxv") pod "d807d28d-b1d2-48c4-9bd4-bb403861afc7" (UID: "d807d28d-b1d2-48c4-9bd4-bb403861afc7"). InnerVolumeSpecName "kube-api-access-sgtxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:41:23.987794 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:23.987761 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgtxv\" (UniqueName: \"kubernetes.io/projected/d807d28d-b1d2-48c4-9bd4-bb403861afc7-kube-api-access-sgtxv\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:41:24.465905 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:24.465869 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d807d28d-b1d2-48c4-9bd4-bb403861afc7" path="/var/lib/kubelet/pods/d807d28d-b1d2-48c4-9bd4-bb403861afc7/volumes" Sep 11 14:41:24.582303 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:24.582269 2578 generic.go:356] "Generic (PLEG): container finished" podID="d807d28d-b1d2-48c4-9bd4-bb403861afc7" containerID="e9a54c44027a19bd48c05ad638793390ae5c15cabc53cdaa0f726b07e9736888" exitCode=137 Sep 11 14:41:24.582767 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:24.582312 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8016-cd8b12" Sep 11 14:41:24.582767 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:24.582358 2578 scope.go:117] "RemoveContainer" containerID="e9a54c44027a19bd48c05ad638793390ae5c15cabc53cdaa0f726b07e9736888" Sep 11 14:41:24.595143 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:24.595122 2578 scope.go:117] "RemoveContainer" containerID="e9a54c44027a19bd48c05ad638793390ae5c15cabc53cdaa0f726b07e9736888" Sep 11 14:41:24.595428 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:41:24.595407 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a54c44027a19bd48c05ad638793390ae5c15cabc53cdaa0f726b07e9736888\": container with ID starting with e9a54c44027a19bd48c05ad638793390ae5c15cabc53cdaa0f726b07e9736888 not found: ID does not exist" containerID="e9a54c44027a19bd48c05ad638793390ae5c15cabc53cdaa0f726b07e9736888" Sep 11 14:41:24.595510 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:24.595437 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a54c44027a19bd48c05ad638793390ae5c15cabc53cdaa0f726b07e9736888"} err="failed to get container status \"e9a54c44027a19bd48c05ad638793390ae5c15cabc53cdaa0f726b07e9736888\": rpc error: code = NotFound desc = could not find container \"e9a54c44027a19bd48c05ad638793390ae5c15cabc53cdaa0f726b07e9736888\": container with ID starting with e9a54c44027a19bd48c05ad638793390ae5c15cabc53cdaa0f726b07e9736888 not found: ID does not exist" Sep 11 14:41:25.729307 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:25.729280 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8016-b1639c" Sep 11 14:41:25.906070 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:25.906036 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9shd\" (UniqueName: \"kubernetes.io/projected/12cff6b4-1879-4541-9b8c-9403fa4ce9f5-kube-api-access-h9shd\") pod \"12cff6b4-1879-4541-9b8c-9403fa4ce9f5\" (UID: \"12cff6b4-1879-4541-9b8c-9403fa4ce9f5\") " Sep 11 14:41:25.908067 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:25.908038 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cff6b4-1879-4541-9b8c-9403fa4ce9f5-kube-api-access-h9shd" (OuterVolumeSpecName: "kube-api-access-h9shd") pod "12cff6b4-1879-4541-9b8c-9403fa4ce9f5" (UID: "12cff6b4-1879-4541-9b8c-9403fa4ce9f5"). InnerVolumeSpecName "kube-api-access-h9shd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:41:26.007838 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:26.007801 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9shd\" (UniqueName: \"kubernetes.io/projected/12cff6b4-1879-4541-9b8c-9403fa4ce9f5-kube-api-access-h9shd\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:41:26.465365 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:26.465329 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12cff6b4-1879-4541-9b8c-9403fa4ce9f5" path="/var/lib/kubelet/pods/12cff6b4-1879-4541-9b8c-9403fa4ce9f5/volumes" Sep 11 14:41:26.597988 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:26.597954 2578 generic.go:356] "Generic (PLEG): container finished" podID="12cff6b4-1879-4541-9b8c-9403fa4ce9f5" containerID="3a93d11189f7a52e62154d61fc72842380399eac67c917bd60b86167cfbe6084" exitCode=137 Sep 11 14:41:26.597988 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:26.597990 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8016-b1639c" Sep 11 14:41:26.598185 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:26.598019 2578 scope.go:117] "RemoveContainer" containerID="3a93d11189f7a52e62154d61fc72842380399eac67c917bd60b86167cfbe6084" Sep 11 14:41:26.607146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:26.607114 2578 scope.go:117] "RemoveContainer" containerID="3a93d11189f7a52e62154d61fc72842380399eac67c917bd60b86167cfbe6084" Sep 11 14:41:26.607424 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:41:26.607404 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a93d11189f7a52e62154d61fc72842380399eac67c917bd60b86167cfbe6084\": container with ID starting with 3a93d11189f7a52e62154d61fc72842380399eac67c917bd60b86167cfbe6084 not found: ID does not exist" containerID="3a93d11189f7a52e62154d61fc72842380399eac67c917bd60b86167cfbe6084" Sep 11 14:41:26.607470 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:26.607432 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a93d11189f7a52e62154d61fc72842380399eac67c917bd60b86167cfbe6084"} err="failed to get container status \"3a93d11189f7a52e62154d61fc72842380399eac67c917bd60b86167cfbe6084\": rpc error: code = NotFound desc = could not find container \"3a93d11189f7a52e62154d61fc72842380399eac67c917bd60b86167cfbe6084\": container with ID starting with 3a93d11189f7a52e62154d61fc72842380399eac67c917bd60b86167cfbe6084 not found: ID does not exist" Sep 11 14:41:36.035764 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:36.035716 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:41:51.227055 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:41:51.227018 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:42:12.611238 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:12.611160 2578 scope.go:117] "RemoveContainer" containerID="79741c22aa1c8fcc80016baa10fc7bd7367847e03fa010871deb72f28e31ec1c" Sep 11 14:42:16.829168 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.829124 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44"] Sep 11 14:42:16.833038 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.830444 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12cff6b4-1879-4541-9b8c-9403fa4ce9f5" containerName="test-curl-8016-b1639c" Sep 11 14:42:16.833038 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.830473 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="12cff6b4-1879-4541-9b8c-9403fa4ce9f5" containerName="test-curl-8016-b1639c" Sep 11 14:42:16.833038 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.830679 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="12cff6b4-1879-4541-9b8c-9403fa4ce9f5" containerName="test-curl-8016-b1639c" Sep 11 14:42:16.836856 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.835383 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:16.839358 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.839334 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ef0ff5-serving-cert\"" Sep 11 14:42:16.846448 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.846421 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44"] Sep 11 14:42:16.868244 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.868215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfd76ae-e408-4591-951b-806e5127020b-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef0ff5-b659c884d-99j44\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:16.868355 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.868263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngvjl\" (UniqueName: \"kubernetes.io/projected/cbfd76ae-e408-4591-951b-806e5127020b-kube-api-access-ngvjl\") pod \"maas-api-e2e-api-a-ef0ff5-b659c884d-99j44\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:16.868413 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.868372 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfd76ae-e408-4591-951b-806e5127020b-metrics-certs\") pod \"maas-api-e2e-api-a-ef0ff5-b659c884d-99j44\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:16.868481 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.868463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfd76ae-e408-4591-951b-806e5127020b-maas-api-tls\") pod \"maas-api-e2e-api-a-ef0ff5-b659c884d-99j44\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:16.969308 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.969264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfd76ae-e408-4591-951b-806e5127020b-metrics-certs\") pod \"maas-api-e2e-api-a-ef0ff5-b659c884d-99j44\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:16.969477 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.969372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfd76ae-e408-4591-951b-806e5127020b-maas-api-tls\") pod \"maas-api-e2e-api-a-ef0ff5-b659c884d-99j44\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:16.969477 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.969447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfd76ae-e408-4591-951b-806e5127020b-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef0ff5-b659c884d-99j44\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:16.969576 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.969499 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngvjl\" (UniqueName: \"kubernetes.io/projected/cbfd76ae-e408-4591-951b-806e5127020b-kube-api-access-ngvjl\") pod \"maas-api-e2e-api-a-ef0ff5-b659c884d-99j44\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:16.970141 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.970112 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfd76ae-e408-4591-951b-806e5127020b-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef0ff5-b659c884d-99j44\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:16.972023 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.971996 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfd76ae-e408-4591-951b-806e5127020b-metrics-certs\") pod \"maas-api-e2e-api-a-ef0ff5-b659c884d-99j44\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:16.972151 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.971996 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfd76ae-e408-4591-951b-806e5127020b-maas-api-tls\") pod \"maas-api-e2e-api-a-ef0ff5-b659c884d-99j44\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:16.979991 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:16.979963 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngvjl\" (UniqueName: \"kubernetes.io/projected/cbfd76ae-e408-4591-951b-806e5127020b-kube-api-access-ngvjl\") pod \"maas-api-e2e-api-a-ef0ff5-b659c884d-99j44\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:17.147627 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:17.147594 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:17.456387 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:17.456361 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44"] Sep 11 14:42:17.457971 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:42:17.457938 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfd76ae_e408_4591_951b_806e5127020b.slice/crio-70a9c932f023374d72ae0eef2b7157c36d82d9db1ec2901094c66b59804571ee WatchSource:0}: Error finding container 70a9c932f023374d72ae0eef2b7157c36d82d9db1ec2901094c66b59804571ee: Status 404 returned error can't find the container with id 70a9c932f023374d72ae0eef2b7157c36d82d9db1ec2901094c66b59804571ee Sep 11 14:42:17.459423 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:17.459404 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:42:17.914328 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:17.914300 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" event={"ID":"cbfd76ae-e408-4591-951b-806e5127020b","Type":"ContainerStarted","Data":"70a9c932f023374d72ae0eef2b7157c36d82d9db1ec2901094c66b59804571ee"} Sep 11 14:42:18.922503 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:18.922470 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" event={"ID":"cbfd76ae-e408-4591-951b-806e5127020b","Type":"ContainerStarted","Data":"583eda2dc652b959b067da2a8d34e639c0cf62af277664135d413071bda1f5ba"} Sep 11 14:42:18.922896 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:18.922596 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:18.954176 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:18.954113 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" podStartSLOduration=2.53997022 podStartE2EDuration="2.954091868s" totalImagesPullingTime="414.121648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:42:18.949792954 +0000 UTC m=+1627.035249316" watchObservedRunningTime="2026-09-11 14:42:18.954091868 +0000 UTC m=+1627.039548230" Sep 11 14:42:22.028164 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:22.028128 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:42:24.933886 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:24.933859 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:42:46.345138 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:46.345101 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:42:48.361246 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.361208 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk"] Sep 11 14:42:48.365336 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.365313 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:48.368253 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.368225 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-56b1b6-serving-cert\"" Sep 11 14:42:48.380883 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.380860 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk"] Sep 11 14:42:48.462392 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.462360 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4132cbff-2d1f-4ae9-acce-900ee49a1526-openshift-service-ca\") pod \"maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:48.462585 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.462405 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-metrics-certs\") pod \"maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:48.462585 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.462493 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-maas-api-tls\") pod \"maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:48.462585 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.462532 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcnf7\" (UniqueName: \"kubernetes.io/projected/4132cbff-2d1f-4ae9-acce-900ee49a1526-kube-api-access-tcnf7\") pod \"maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:48.563607 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.563573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-metrics-certs\") pod \"maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:48.563816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.563688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-maas-api-tls\") pod \"maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:48.563816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.563782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcnf7\" (UniqueName: \"kubernetes.io/projected/4132cbff-2d1f-4ae9-acce-900ee49a1526-kube-api-access-tcnf7\") pod \"maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:48.563963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.563919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4132cbff-2d1f-4ae9-acce-900ee49a1526-openshift-service-ca\") pod \"maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:48.564017 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:42:48.563963 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-serving-cert: secret "maas-api-e2e-api-b-56b1b6-serving-cert" not found Sep 11 14:42:48.564069 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:42:48.564057 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-maas-api-tls podName:4132cbff-2d1f-4ae9-acce-900ee49a1526 nodeName:}" failed. No retries permitted until 2026-09-11 14:42:49.064034926 +0000 UTC m=+1657.149491279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-maas-api-tls") pod "maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" (UID: "4132cbff-2d1f-4ae9-acce-900ee49a1526") : secret "maas-api-e2e-api-b-56b1b6-serving-cert" not found Sep 11 14:42:48.564681 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.564659 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4132cbff-2d1f-4ae9-acce-900ee49a1526-openshift-service-ca\") pod \"maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:48.566092 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.566071 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-metrics-certs\") pod \"maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:48.582326 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:48.582300 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcnf7\" (UniqueName: \"kubernetes.io/projected/4132cbff-2d1f-4ae9-acce-900ee49a1526-kube-api-access-tcnf7\") pod \"maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:49.069524 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:49.069484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-maas-api-tls\") pod \"maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:49.071922 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:49.071886 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-maas-api-tls\") pod \"maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:49.280428 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:49.280392 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:49.564779 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:49.564748 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk"] Sep 11 14:42:49.565633 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:42:49.565605 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4132cbff_2d1f_4ae9_acce_900ee49a1526.slice/crio-7c38943ee3aeacbb35be50b2130e5356fd1c44984e1121dcd777c3b5c141ddcd WatchSource:0}: Error finding container 7c38943ee3aeacbb35be50b2130e5356fd1c44984e1121dcd777c3b5c141ddcd: Status 404 returned error can't find the container with id 7c38943ee3aeacbb35be50b2130e5356fd1c44984e1121dcd777c3b5c141ddcd Sep 11 14:42:50.117106 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:50.117060 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" event={"ID":"4132cbff-2d1f-4ae9-acce-900ee49a1526","Type":"ContainerStarted","Data":"32c6ce986f3ecd4ed81d77a7941ea3a8c4bf3b48f8eb3fbfa271a0c99396d22a"} Sep 11 14:42:50.117289 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:50.117124 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:42:50.117289 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:50.117142 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" event={"ID":"4132cbff-2d1f-4ae9-acce-900ee49a1526","Type":"ContainerStarted","Data":"7c38943ee3aeacbb35be50b2130e5356fd1c44984e1121dcd777c3b5c141ddcd"} Sep 11 14:42:50.154516 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:50.154405 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" podStartSLOduration=1.844171223 podStartE2EDuration="2.154387839s" totalImagesPullingTime="310.216616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:42:50.151213347 +0000 UTC m=+1658.236669721" watchObservedRunningTime="2026-09-11 14:42:50.154387839 +0000 UTC m=+1658.239844236" Sep 11 14:42:56.128813 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:42:56.128784 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:43:08.124436 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:08.124391 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:43:20.020124 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:20.020091 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:43:24.549616 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.549581 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh"] Sep 11 14:43:24.555868 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.555819 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:24.559050 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.559028 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f0ee6c-serving-cert\"" Sep 11 14:43:24.567316 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.567253 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh"] Sep 11 14:43:24.719919 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.719885 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6sb4\" (UniqueName: \"kubernetes.io/projected/600ad8ea-9f04-4b4d-9206-57e9a46a202f-kube-api-access-m6sb4\") pod \"maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:24.720102 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.719945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600ad8ea-9f04-4b4d-9206-57e9a46a202f-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:24.720102 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.719998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600ad8ea-9f04-4b4d-9206-57e9a46a202f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:24.720102 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.720053 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600ad8ea-9f04-4b4d-9206-57e9a46a202f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:24.821388 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.821316 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600ad8ea-9f04-4b4d-9206-57e9a46a202f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:24.821531 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.821411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600ad8ea-9f04-4b4d-9206-57e9a46a202f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:24.821531 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.821490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6sb4\" (UniqueName: \"kubernetes.io/projected/600ad8ea-9f04-4b4d-9206-57e9a46a202f-kube-api-access-m6sb4\") pod \"maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:24.821624 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.821547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600ad8ea-9f04-4b4d-9206-57e9a46a202f-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:24.822185 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.822161 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600ad8ea-9f04-4b4d-9206-57e9a46a202f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:24.823900 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.823879 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600ad8ea-9f04-4b4d-9206-57e9a46a202f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:24.824018 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.824000 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600ad8ea-9f04-4b4d-9206-57e9a46a202f-metrics-certs\") pod \"maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:24.842909 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.842885 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6sb4\" (UniqueName: \"kubernetes.io/projected/600ad8ea-9f04-4b4d-9206-57e9a46a202f-kube-api-access-m6sb4\") pod \"maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:24.869396 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:24.869374 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:25.146695 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:25.146664 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh"] Sep 11 14:43:25.147301 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:43:25.147276 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600ad8ea_9f04_4b4d_9206_57e9a46a202f.slice/crio-b8347bc0f67b897212df9fdc498eedf52b79d0d097ca56c2542d5a71618d21bb WatchSource:0}: Error finding container b8347bc0f67b897212df9fdc498eedf52b79d0d097ca56c2542d5a71618d21bb: Status 404 returned error can't find the container with id b8347bc0f67b897212df9fdc498eedf52b79d0d097ca56c2542d5a71618d21bb Sep 11 14:43:25.335453 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:25.335419 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" event={"ID":"600ad8ea-9f04-4b4d-9206-57e9a46a202f","Type":"ContainerStarted","Data":"b8347bc0f67b897212df9fdc498eedf52b79d0d097ca56c2542d5a71618d21bb"} Sep 11 14:43:26.343977 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:26.343941 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" event={"ID":"600ad8ea-9f04-4b4d-9206-57e9a46a202f","Type":"ContainerStarted","Data":"8038690ea7ae077249421c8ec8daccb2b801f36f01469bab323a57ee640e38d6"} Sep 11 14:43:26.344358 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:26.344000 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:26.367243 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:26.367191 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" podStartSLOduration=2.024438873 podStartE2EDuration="2.367176589s" totalImagesPullingTime="342.737716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:43:26.365092822 +0000 UTC m=+1694.450549184" watchObservedRunningTime="2026-09-11 14:43:26.367176589 +0000 UTC m=+1694.452632971" Sep 11 14:43:32.357389 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:32.357352 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:43:33.248947 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:33.248905 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:43:33.958482 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:33.958444 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44"] Sep 11 14:43:33.958998 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:33.958686 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" podUID="cbfd76ae-e408-4591-951b-806e5127020b" containerName="maas-api" containerID="cri-o://583eda2dc652b959b067da2a8d34e639c0cf62af277664135d413071bda1f5ba" gracePeriod=30 Sep 11 14:43:34.222042 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.221816 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:43:34.396033 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.395996 2578 generic.go:356] "Generic (PLEG): container finished" podID="cbfd76ae-e408-4591-951b-806e5127020b" containerID="583eda2dc652b959b067da2a8d34e639c0cf62af277664135d413071bda1f5ba" exitCode=0 Sep 11 14:43:34.396195 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.396045 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" Sep 11 14:43:34.396195 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.396089 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" event={"ID":"cbfd76ae-e408-4591-951b-806e5127020b","Type":"ContainerDied","Data":"583eda2dc652b959b067da2a8d34e639c0cf62af277664135d413071bda1f5ba"} Sep 11 14:43:34.396195 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.396148 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44" event={"ID":"cbfd76ae-e408-4591-951b-806e5127020b","Type":"ContainerDied","Data":"70a9c932f023374d72ae0eef2b7157c36d82d9db1ec2901094c66b59804571ee"} Sep 11 14:43:34.396195 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.396179 2578 scope.go:117] "RemoveContainer" containerID="583eda2dc652b959b067da2a8d34e639c0cf62af277664135d413071bda1f5ba" Sep 11 14:43:34.405495 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.405478 2578 scope.go:117] "RemoveContainer" containerID="583eda2dc652b959b067da2a8d34e639c0cf62af277664135d413071bda1f5ba" Sep 11 14:43:34.405751 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:43:34.405720 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"583eda2dc652b959b067da2a8d34e639c0cf62af277664135d413071bda1f5ba\": container with ID starting with 583eda2dc652b959b067da2a8d34e639c0cf62af277664135d413071bda1f5ba not found: ID does not exist" containerID="583eda2dc652b959b067da2a8d34e639c0cf62af277664135d413071bda1f5ba" Sep 11 14:43:34.405804 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.405756 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"583eda2dc652b959b067da2a8d34e639c0cf62af277664135d413071bda1f5ba"} err="failed to get container status \"583eda2dc652b959b067da2a8d34e639c0cf62af277664135d413071bda1f5ba\": rpc error: code = NotFound desc = could not find container \"583eda2dc652b959b067da2a8d34e639c0cf62af277664135d413071bda1f5ba\": container with ID starting with 583eda2dc652b959b067da2a8d34e639c0cf62af277664135d413071bda1f5ba not found: ID does not exist" Sep 11 14:43:34.411134 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.411115 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfd76ae-e408-4591-951b-806e5127020b-maas-api-tls\") pod \"cbfd76ae-e408-4591-951b-806e5127020b\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " Sep 11 14:43:34.411203 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.411160 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngvjl\" (UniqueName: \"kubernetes.io/projected/cbfd76ae-e408-4591-951b-806e5127020b-kube-api-access-ngvjl\") pod \"cbfd76ae-e408-4591-951b-806e5127020b\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " Sep 11 14:43:34.411203 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.411179 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfd76ae-e408-4591-951b-806e5127020b-openshift-service-ca\") pod \"cbfd76ae-e408-4591-951b-806e5127020b\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " Sep 11 14:43:34.411335 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.411321 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfd76ae-e408-4591-951b-806e5127020b-metrics-certs\") pod \"cbfd76ae-e408-4591-951b-806e5127020b\" (UID: \"cbfd76ae-e408-4591-951b-806e5127020b\") " Sep 11 14:43:34.411573 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.411547 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbfd76ae-e408-4591-951b-806e5127020b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cbfd76ae-e408-4591-951b-806e5127020b" (UID: "cbfd76ae-e408-4591-951b-806e5127020b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:43:34.411695 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.411657 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbfd76ae-e408-4591-951b-806e5127020b-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:43:34.413183 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.413155 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbfd76ae-e408-4591-951b-806e5127020b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cbfd76ae-e408-4591-951b-806e5127020b" (UID: "cbfd76ae-e408-4591-951b-806e5127020b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:43:34.413282 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.413189 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbfd76ae-e408-4591-951b-806e5127020b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cbfd76ae-e408-4591-951b-806e5127020b" (UID: "cbfd76ae-e408-4591-951b-806e5127020b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:43:34.413282 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.413249 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfd76ae-e408-4591-951b-806e5127020b-kube-api-access-ngvjl" (OuterVolumeSpecName: "kube-api-access-ngvjl") pod "cbfd76ae-e408-4591-951b-806e5127020b" (UID: "cbfd76ae-e408-4591-951b-806e5127020b"). InnerVolumeSpecName "kube-api-access-ngvjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:43:34.512936 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.512903 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfd76ae-e408-4591-951b-806e5127020b-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:43:34.512936 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.512939 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbfd76ae-e408-4591-951b-806e5127020b-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:43:34.513119 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.512953 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngvjl\" (UniqueName: \"kubernetes.io/projected/cbfd76ae-e408-4591-951b-806e5127020b-kube-api-access-ngvjl\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:43:34.718858 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.718824 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44"] Sep 11 14:43:34.727090 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:34.727015 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef0ff5-b659c884d-99j44"] Sep 11 14:43:36.465746 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:36.465708 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbfd76ae-e408-4591-951b-806e5127020b" path="/var/lib/kubelet/pods/cbfd76ae-e408-4591-951b-806e5127020b/volumes" Sep 11 14:43:48.728153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:48.728110 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:43:52.570440 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.570405 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq"] Sep 11 14:43:52.570941 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.570907 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbfd76ae-e408-4591-951b-806e5127020b" containerName="maas-api" Sep 11 14:43:52.570941 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.570921 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbfd76ae-e408-4591-951b-806e5127020b" containerName="maas-api" Sep 11 14:43:52.571040 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.571027 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbfd76ae-e408-4591-951b-806e5127020b" containerName="maas-api" Sep 11 14:43:52.574600 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.574581 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:52.578662 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.578644 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-40d3a2-serving-cert\"" Sep 11 14:43:52.609485 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.609464 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq"] Sep 11 14:43:52.677309 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.677282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jvsj\" (UniqueName: \"kubernetes.io/projected/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-kube-api-access-5jvsj\") pod \"maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:52.677456 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.677320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-metrics-certs\") pod \"maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:52.677456 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.677352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:52.677574 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.677475 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:52.778747 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.778694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jvsj\" (UniqueName: \"kubernetes.io/projected/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-kube-api-access-5jvsj\") pod \"maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:52.778908 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.778766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-metrics-certs\") pod \"maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:52.778908 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.778820 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:52.778908 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.778895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:52.779038 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:43:52.779021 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-serving-cert: secret "maas-api-e2e-ipp-b-40d3a2-serving-cert" not found Sep 11 14:43:52.779097 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:43:52.779087 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-maas-api-tls podName:7ee3df4d-bd69-4159-88a1-52329a7e8f1e nodeName:}" failed. No retries permitted until 2026-09-11 14:43:53.279070433 +0000 UTC m=+1721.364526776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-maas-api-tls") pod "maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" (UID: "7ee3df4d-bd69-4159-88a1-52329a7e8f1e") : secret "maas-api-e2e-ipp-b-40d3a2-serving-cert" not found Sep 11 14:43:52.779521 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.779489 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:52.781264 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.781243 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-metrics-certs\") pod \"maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:52.793096 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:52.793068 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jvsj\" (UniqueName: \"kubernetes.io/projected/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-kube-api-access-5jvsj\") pod \"maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:53.283705 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:53.283667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:53.286121 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:53.286099 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:53.484375 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:53.484343 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:53.772961 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:53.772934 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq"] Sep 11 14:43:53.773997 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:43:53.773970 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee3df4d_bd69_4159_88a1_52329a7e8f1e.slice/crio-e4a245247a2a50d31d6692e5297f9ac45b7ca830072aa2e60b266568e35bd962 WatchSource:0}: Error finding container e4a245247a2a50d31d6692e5297f9ac45b7ca830072aa2e60b266568e35bd962: Status 404 returned error can't find the container with id e4a245247a2a50d31d6692e5297f9ac45b7ca830072aa2e60b266568e35bd962 Sep 11 14:43:54.535844 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:54.535759 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" event={"ID":"7ee3df4d-bd69-4159-88a1-52329a7e8f1e","Type":"ContainerStarted","Data":"2f88769607b7380fdbb6e22585642f730dacdea521709c5b7dd2dea9e5e5dd18"} Sep 11 14:43:54.535844 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:54.535822 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:43:54.536023 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:54.535847 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" event={"ID":"7ee3df4d-bd69-4159-88a1-52329a7e8f1e","Type":"ContainerStarted","Data":"e4a245247a2a50d31d6692e5297f9ac45b7ca830072aa2e60b266568e35bd962"} Sep 11 14:43:54.558011 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:54.557954 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" podStartSLOduration=2.194854474 podStartE2EDuration="2.557936088s" totalImagesPullingTime="363.081614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:43:54.55606377 +0000 UTC m=+1722.641520131" watchObservedRunningTime="2026-09-11 14:43:54.557936088 +0000 UTC m=+1722.643392451" Sep 11 14:43:57.539123 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:43:57.539087 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:44:00.547768 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:00.547739 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:44:01.756496 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:01.756450 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk"] Sep 11 14:44:01.756929 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:01.756773 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" podUID="4132cbff-2d1f-4ae9-acce-900ee49a1526" containerName="maas-api" containerID="cri-o://32c6ce986f3ecd4ed81d77a7941ea3a8c4bf3b48f8eb3fbfa271a0c99396d22a" gracePeriod=30 Sep 11 14:44:02.037464 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.037440 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:44:02.169418 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.169385 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4132cbff-2d1f-4ae9-acce-900ee49a1526-openshift-service-ca\") pod \"4132cbff-2d1f-4ae9-acce-900ee49a1526\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " Sep 11 14:44:02.169603 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.169467 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcnf7\" (UniqueName: \"kubernetes.io/projected/4132cbff-2d1f-4ae9-acce-900ee49a1526-kube-api-access-tcnf7\") pod \"4132cbff-2d1f-4ae9-acce-900ee49a1526\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " Sep 11 14:44:02.169603 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.169509 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-metrics-certs\") pod \"4132cbff-2d1f-4ae9-acce-900ee49a1526\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " Sep 11 14:44:02.169603 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.169547 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-maas-api-tls\") pod \"4132cbff-2d1f-4ae9-acce-900ee49a1526\" (UID: \"4132cbff-2d1f-4ae9-acce-900ee49a1526\") " Sep 11 14:44:02.169812 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.169784 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4132cbff-2d1f-4ae9-acce-900ee49a1526-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4132cbff-2d1f-4ae9-acce-900ee49a1526" (UID: "4132cbff-2d1f-4ae9-acce-900ee49a1526"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:02.169940 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.169922 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4132cbff-2d1f-4ae9-acce-900ee49a1526-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:02.171612 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.171587 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4132cbff-2d1f-4ae9-acce-900ee49a1526" (UID: "4132cbff-2d1f-4ae9-acce-900ee49a1526"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:44:02.171612 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.171607 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4132cbff-2d1f-4ae9-acce-900ee49a1526" (UID: "4132cbff-2d1f-4ae9-acce-900ee49a1526"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:44:02.171782 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.171685 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4132cbff-2d1f-4ae9-acce-900ee49a1526-kube-api-access-tcnf7" (OuterVolumeSpecName: "kube-api-access-tcnf7") pod "4132cbff-2d1f-4ae9-acce-900ee49a1526" (UID: "4132cbff-2d1f-4ae9-acce-900ee49a1526"). InnerVolumeSpecName "kube-api-access-tcnf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:02.271161 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.271084 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcnf7\" (UniqueName: \"kubernetes.io/projected/4132cbff-2d1f-4ae9-acce-900ee49a1526-kube-api-access-tcnf7\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:02.271161 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.271114 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:02.271161 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.271123 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4132cbff-2d1f-4ae9-acce-900ee49a1526-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:02.592855 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.592777 2578 generic.go:356] "Generic (PLEG): container finished" podID="4132cbff-2d1f-4ae9-acce-900ee49a1526" containerID="32c6ce986f3ecd4ed81d77a7941ea3a8c4bf3b48f8eb3fbfa271a0c99396d22a" exitCode=0 Sep 11 14:44:02.592855 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.592824 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" Sep 11 14:44:02.593077 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.592864 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" event={"ID":"4132cbff-2d1f-4ae9-acce-900ee49a1526","Type":"ContainerDied","Data":"32c6ce986f3ecd4ed81d77a7941ea3a8c4bf3b48f8eb3fbfa271a0c99396d22a"} Sep 11 14:44:02.593077 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.592920 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk" event={"ID":"4132cbff-2d1f-4ae9-acce-900ee49a1526","Type":"ContainerDied","Data":"7c38943ee3aeacbb35be50b2130e5356fd1c44984e1121dcd777c3b5c141ddcd"} Sep 11 14:44:02.593077 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.592951 2578 scope.go:117] "RemoveContainer" containerID="32c6ce986f3ecd4ed81d77a7941ea3a8c4bf3b48f8eb3fbfa271a0c99396d22a" Sep 11 14:44:02.602343 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.602324 2578 scope.go:117] "RemoveContainer" containerID="32c6ce986f3ecd4ed81d77a7941ea3a8c4bf3b48f8eb3fbfa271a0c99396d22a" Sep 11 14:44:02.602606 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:44:02.602590 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c6ce986f3ecd4ed81d77a7941ea3a8c4bf3b48f8eb3fbfa271a0c99396d22a\": container with ID starting with 32c6ce986f3ecd4ed81d77a7941ea3a8c4bf3b48f8eb3fbfa271a0c99396d22a not found: ID does not exist" containerID="32c6ce986f3ecd4ed81d77a7941ea3a8c4bf3b48f8eb3fbfa271a0c99396d22a" Sep 11 14:44:02.602646 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.602622 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c6ce986f3ecd4ed81d77a7941ea3a8c4bf3b48f8eb3fbfa271a0c99396d22a"} err="failed to get container status \"32c6ce986f3ecd4ed81d77a7941ea3a8c4bf3b48f8eb3fbfa271a0c99396d22a\": rpc error: code = NotFound desc = could not find container \"32c6ce986f3ecd4ed81d77a7941ea3a8c4bf3b48f8eb3fbfa271a0c99396d22a\": container with ID starting with 32c6ce986f3ecd4ed81d77a7941ea3a8c4bf3b48f8eb3fbfa271a0c99396d22a not found: ID does not exist" Sep 11 14:44:02.615614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.615591 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk"] Sep 11 14:44:02.631671 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:02.631646 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56b1b6-76f4c965d7-b9tqk"] Sep 11 14:44:04.467937 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:04.467899 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4132cbff-2d1f-4ae9-acce-900ee49a1526" path="/var/lib/kubelet/pods/4132cbff-2d1f-4ae9-acce-900ee49a1526/volumes" Sep 11 14:44:14.834236 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:14.834200 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:44:17.324995 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:17.324961 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jjtl7_bb207b52-67df-47fb-a509-b099fafc0120/payload-processing/0.log" Sep 11 14:44:23.536712 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.536662 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc"] Sep 11 14:44:23.537300 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.536999 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" podUID="b940871f-8c89-477c-9755-6f3f54e096df" containerName="maas-api" containerID="cri-o://426d97bb91587a8781eed824c4ac2c1200349985cd0756c24caf2c13095c8adc" gracePeriod=30 Sep 11 14:44:23.724302 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.724191 2578 generic.go:356] "Generic (PLEG): container finished" podID="b940871f-8c89-477c-9755-6f3f54e096df" containerID="426d97bb91587a8781eed824c4ac2c1200349985cd0756c24caf2c13095c8adc" exitCode=0 Sep 11 14:44:23.724302 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.724226 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" event={"ID":"b940871f-8c89-477c-9755-6f3f54e096df","Type":"ContainerDied","Data":"426d97bb91587a8781eed824c4ac2c1200349985cd0756c24caf2c13095c8adc"} Sep 11 14:44:23.807811 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.807789 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:44:23.966550 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.966516 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-metrics-certs\") pod \"b940871f-8c89-477c-9755-6f3f54e096df\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " Sep 11 14:44:23.966751 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.966597 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8k5f\" (UniqueName: \"kubernetes.io/projected/b940871f-8c89-477c-9755-6f3f54e096df-kube-api-access-r8k5f\") pod \"b940871f-8c89-477c-9755-6f3f54e096df\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " Sep 11 14:44:23.966751 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.966648 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b940871f-8c89-477c-9755-6f3f54e096df-openshift-service-ca\") pod \"b940871f-8c89-477c-9755-6f3f54e096df\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " Sep 11 14:44:23.966751 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.966707 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-maas-api-tls\") pod \"b940871f-8c89-477c-9755-6f3f54e096df\" (UID: \"b940871f-8c89-477c-9755-6f3f54e096df\") " Sep 11 14:44:23.967008 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.966982 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b940871f-8c89-477c-9755-6f3f54e096df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b940871f-8c89-477c-9755-6f3f54e096df" (UID: "b940871f-8c89-477c-9755-6f3f54e096df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:23.967087 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.967070 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b940871f-8c89-477c-9755-6f3f54e096df-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:23.968768 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.968714 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b940871f-8c89-477c-9755-6f3f54e096df-kube-api-access-r8k5f" (OuterVolumeSpecName: "kube-api-access-r8k5f") pod "b940871f-8c89-477c-9755-6f3f54e096df" (UID: "b940871f-8c89-477c-9755-6f3f54e096df"). InnerVolumeSpecName "kube-api-access-r8k5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:23.968768 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.968723 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b940871f-8c89-477c-9755-6f3f54e096df" (UID: "b940871f-8c89-477c-9755-6f3f54e096df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:44:23.968915 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:23.968823 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b940871f-8c89-477c-9755-6f3f54e096df" (UID: "b940871f-8c89-477c-9755-6f3f54e096df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:44:24.068325 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:24.068237 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:24.068325 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:24.068267 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b940871f-8c89-477c-9755-6f3f54e096df-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:24.068325 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:24.068280 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8k5f\" (UniqueName: \"kubernetes.io/projected/b940871f-8c89-477c-9755-6f3f54e096df-kube-api-access-r8k5f\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:24.731800 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:24.731688 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" event={"ID":"b940871f-8c89-477c-9755-6f3f54e096df","Type":"ContainerDied","Data":"3be721ce33420c6dfe399e5ee424744fa30339f0cca1f81973a73a168f80e318"} Sep 11 14:44:24.731800 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:24.731776 2578 scope.go:117] "RemoveContainer" containerID="426d97bb91587a8781eed824c4ac2c1200349985cd0756c24caf2c13095c8adc" Sep 11 14:44:24.731800 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:24.731710 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc" Sep 11 14:44:24.767435 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:24.767407 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc"] Sep 11 14:44:24.771573 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:24.771551 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-958398-7c6bbd5987-txmwc"] Sep 11 14:44:26.464342 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:26.464309 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b940871f-8c89-477c-9755-6f3f54e096df" path="/var/lib/kubelet/pods/b940871f-8c89-477c-9755-6f3f54e096df/volumes" Sep 11 14:44:31.878611 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:31.878571 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv"] Sep 11 14:44:31.879096 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:31.878872 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" containerID="cri-o://66d0dbb53ef661a64d71ace54d3679239047d76f5ee93ef058ef91b7fb614f97" gracePeriod=30 Sep 11 14:44:33.835533 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:33.835499 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:44:35.416692 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:35.416652 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:44:35.417064 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:35.416720 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:44:37.137377 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.137355 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:44:37.183866 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.183810 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-data\") pod \"1e553792-1de7-4dd9-b43f-d22976b0d94f\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " Sep 11 14:44:37.183866 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.183845 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-workload-certs\") pod \"1e553792-1de7-4dd9-b43f-d22976b0d94f\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " Sep 11 14:44:37.184037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.183898 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-credential-socket\") pod \"1e553792-1de7-4dd9-b43f-d22976b0d94f\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " Sep 11 14:44:37.184037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.183933 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-podinfo\") pod \"1e553792-1de7-4dd9-b43f-d22976b0d94f\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " Sep 11 14:44:37.184037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.183951 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-workload-socket\") pod \"1e553792-1de7-4dd9-b43f-d22976b0d94f\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " Sep 11 14:44:37.184037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.183968 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-envoy\") pod \"1e553792-1de7-4dd9-b43f-d22976b0d94f\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " Sep 11 14:44:37.184037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.183998 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8kdj\" (UniqueName: \"kubernetes.io/projected/1e553792-1de7-4dd9-b43f-d22976b0d94f-kube-api-access-l8kdj\") pod \"1e553792-1de7-4dd9-b43f-d22976b0d94f\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " Sep 11 14:44:37.184037 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.184025 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-token\") pod \"1e553792-1de7-4dd9-b43f-d22976b0d94f\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " Sep 11 14:44:37.184342 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.184095 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e553792-1de7-4dd9-b43f-d22976b0d94f-istiod-ca-cert\") pod \"1e553792-1de7-4dd9-b43f-d22976b0d94f\" (UID: \"1e553792-1de7-4dd9-b43f-d22976b0d94f\") " Sep 11 14:44:37.184342 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.184087 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e553792-1de7-4dd9-b43f-d22976b0d94f" (UID: "1e553792-1de7-4dd9-b43f-d22976b0d94f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:37.184342 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.184156 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e553792-1de7-4dd9-b43f-d22976b0d94f" (UID: "1e553792-1de7-4dd9-b43f-d22976b0d94f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:37.184342 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.184276 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e553792-1de7-4dd9-b43f-d22976b0d94f" (UID: "1e553792-1de7-4dd9-b43f-d22976b0d94f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:37.184534 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.184386 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e553792-1de7-4dd9-b43f-d22976b0d94f" (UID: "1e553792-1de7-4dd9-b43f-d22976b0d94f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:37.184534 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.184430 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-data\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:37.184534 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.184451 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-workload-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:37.184534 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.184466 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-credential-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:37.184534 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.184482 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e553792-1de7-4dd9-b43f-d22976b0d94f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e553792-1de7-4dd9-b43f-d22976b0d94f" (UID: "1e553792-1de7-4dd9-b43f-d22976b0d94f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:37.186096 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.186071 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e553792-1de7-4dd9-b43f-d22976b0d94f" (UID: "1e553792-1de7-4dd9-b43f-d22976b0d94f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:44:37.186213 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.186183 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e553792-1de7-4dd9-b43f-d22976b0d94f" (UID: "1e553792-1de7-4dd9-b43f-d22976b0d94f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:37.186213 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.186197 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e553792-1de7-4dd9-b43f-d22976b0d94f" (UID: "1e553792-1de7-4dd9-b43f-d22976b0d94f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:37.186330 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.186307 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e553792-1de7-4dd9-b43f-d22976b0d94f-kube-api-access-l8kdj" (OuterVolumeSpecName: "kube-api-access-l8kdj") pod "1e553792-1de7-4dd9-b43f-d22976b0d94f" (UID: "1e553792-1de7-4dd9-b43f-d22976b0d94f"). InnerVolumeSpecName "kube-api-access-l8kdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:37.286354 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.286277 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-podinfo\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:37.286354 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.286311 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-workload-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:37.286354 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.286334 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-envoy\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:37.286645 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.286622 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8kdj\" (UniqueName: \"kubernetes.io/projected/1e553792-1de7-4dd9-b43f-d22976b0d94f-kube-api-access-l8kdj\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:37.286794 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.286647 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e553792-1de7-4dd9-b43f-d22976b0d94f-istio-token\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:37.286794 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.286663 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e553792-1de7-4dd9-b43f-d22976b0d94f-istiod-ca-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:37.819599 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.819561 2578 generic.go:356] "Generic (PLEG): container finished" podID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerID="66d0dbb53ef661a64d71ace54d3679239047d76f5ee93ef058ef91b7fb614f97" exitCode=0 Sep 11 14:44:37.819782 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.819622 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" Sep 11 14:44:37.819782 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.819638 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" event={"ID":"1e553792-1de7-4dd9-b43f-d22976b0d94f","Type":"ContainerDied","Data":"66d0dbb53ef661a64d71ace54d3679239047d76f5ee93ef058ef91b7fb614f97"} Sep 11 14:44:37.819782 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.819696 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv" event={"ID":"1e553792-1de7-4dd9-b43f-d22976b0d94f","Type":"ContainerDied","Data":"87944d8a952ff0b7d9ea1ed71cf57073222069026c3575b316d48732d032312e"} Sep 11 14:44:37.819782 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.819722 2578 scope.go:117] "RemoveContainer" containerID="66d0dbb53ef661a64d71ace54d3679239047d76f5ee93ef058ef91b7fb614f97" Sep 11 14:44:37.829665 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.829647 2578 scope.go:117] "RemoveContainer" containerID="e0f2a81b3a4e3ae4994bbdb9c1f19838399dbce115fef879e89628b7bc1e5643" Sep 11 14:44:37.837678 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.837663 2578 scope.go:117] "RemoveContainer" containerID="66d0dbb53ef661a64d71ace54d3679239047d76f5ee93ef058ef91b7fb614f97" Sep 11 14:44:37.837956 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:44:37.837937 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d0dbb53ef661a64d71ace54d3679239047d76f5ee93ef058ef91b7fb614f97\": container with ID starting with 66d0dbb53ef661a64d71ace54d3679239047d76f5ee93ef058ef91b7fb614f97 not found: ID does not exist" containerID="66d0dbb53ef661a64d71ace54d3679239047d76f5ee93ef058ef91b7fb614f97" Sep 11 14:44:37.838011 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.837963 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d0dbb53ef661a64d71ace54d3679239047d76f5ee93ef058ef91b7fb614f97"} err="failed to get container status \"66d0dbb53ef661a64d71ace54d3679239047d76f5ee93ef058ef91b7fb614f97\": rpc error: code = NotFound desc = could not find container \"66d0dbb53ef661a64d71ace54d3679239047d76f5ee93ef058ef91b7fb614f97\": container with ID starting with 66d0dbb53ef661a64d71ace54d3679239047d76f5ee93ef058ef91b7fb614f97 not found: ID does not exist" Sep 11 14:44:37.838011 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.837980 2578 scope.go:117] "RemoveContainer" containerID="e0f2a81b3a4e3ae4994bbdb9c1f19838399dbce115fef879e89628b7bc1e5643" Sep 11 14:44:37.838223 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:44:37.838204 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f2a81b3a4e3ae4994bbdb9c1f19838399dbce115fef879e89628b7bc1e5643\": container with ID starting with e0f2a81b3a4e3ae4994bbdb9c1f19838399dbce115fef879e89628b7bc1e5643 not found: ID does not exist" containerID="e0f2a81b3a4e3ae4994bbdb9c1f19838399dbce115fef879e89628b7bc1e5643" Sep 11 14:44:37.838265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:37.838230 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f2a81b3a4e3ae4994bbdb9c1f19838399dbce115fef879e89628b7bc1e5643"} err="failed to get container status \"e0f2a81b3a4e3ae4994bbdb9c1f19838399dbce115fef879e89628b7bc1e5643\": rpc error: code = NotFound desc = could not find container \"e0f2a81b3a4e3ae4994bbdb9c1f19838399dbce115fef879e89628b7bc1e5643\": container with ID starting with e0f2a81b3a4e3ae4994bbdb9c1f19838399dbce115fef879e89628b7bc1e5643 not found: ID does not exist" Sep 11 14:44:39.300998 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:39.300966 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv"] Sep 11 14:44:39.305241 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:39.305213 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-958398-openshift-default-644fb4794-298nv"] Sep 11 14:44:40.464994 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:40.464962 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" path="/var/lib/kubelet/pods/1e553792-1de7-4dd9-b43f-d22976b0d94f/volumes" Sep 11 14:44:43.360918 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.360884 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw"] Sep 11 14:44:43.361318 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.361150 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" podUID="f5f0985b-b8f7-4140-9870-5a481665636b" containerName="maas-api" containerID="cri-o://532d1f44cfb09070e2e5ddd86f04e66f5caeb1d1887ce25532cb6eb98aeee717" gracePeriod=30 Sep 11 14:44:43.613595 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.613539 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:44:43.744318 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.744283 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlq76\" (UniqueName: \"kubernetes.io/projected/f5f0985b-b8f7-4140-9870-5a481665636b-kube-api-access-hlq76\") pod \"f5f0985b-b8f7-4140-9870-5a481665636b\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " Sep 11 14:44:43.744318 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.744321 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f0985b-b8f7-4140-9870-5a481665636b-openshift-service-ca\") pod \"f5f0985b-b8f7-4140-9870-5a481665636b\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " Sep 11 14:44:43.744554 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.744345 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-metrics-certs\") pod \"f5f0985b-b8f7-4140-9870-5a481665636b\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " Sep 11 14:44:43.744554 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.744442 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-maas-api-tls\") pod \"f5f0985b-b8f7-4140-9870-5a481665636b\" (UID: \"f5f0985b-b8f7-4140-9870-5a481665636b\") " Sep 11 14:44:43.744768 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.744711 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f0985b-b8f7-4140-9870-5a481665636b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f5f0985b-b8f7-4140-9870-5a481665636b" (UID: "f5f0985b-b8f7-4140-9870-5a481665636b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:43.746630 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.746596 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f5f0985b-b8f7-4140-9870-5a481665636b" (UID: "f5f0985b-b8f7-4140-9870-5a481665636b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:44:43.746630 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.746621 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f5f0985b-b8f7-4140-9870-5a481665636b" (UID: "f5f0985b-b8f7-4140-9870-5a481665636b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:44:43.746797 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.746596 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f0985b-b8f7-4140-9870-5a481665636b-kube-api-access-hlq76" (OuterVolumeSpecName: "kube-api-access-hlq76") pod "f5f0985b-b8f7-4140-9870-5a481665636b" (UID: "f5f0985b-b8f7-4140-9870-5a481665636b"). InnerVolumeSpecName "kube-api-access-hlq76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:43.845789 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.845750 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlq76\" (UniqueName: \"kubernetes.io/projected/f5f0985b-b8f7-4140-9870-5a481665636b-kube-api-access-hlq76\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:43.845789 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.845783 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f0985b-b8f7-4140-9870-5a481665636b-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:43.845789 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.845796 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:43.846029 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.845808 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5f0985b-b8f7-4140-9870-5a481665636b-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:43.858417 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.858388 2578 generic.go:356] "Generic (PLEG): container finished" podID="f5f0985b-b8f7-4140-9870-5a481665636b" containerID="532d1f44cfb09070e2e5ddd86f04e66f5caeb1d1887ce25532cb6eb98aeee717" exitCode=0 Sep 11 14:44:43.858605 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.858433 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" Sep 11 14:44:43.858605 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.858476 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" event={"ID":"f5f0985b-b8f7-4140-9870-5a481665636b","Type":"ContainerDied","Data":"532d1f44cfb09070e2e5ddd86f04e66f5caeb1d1887ce25532cb6eb98aeee717"} Sep 11 14:44:43.858605 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.858539 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw" event={"ID":"f5f0985b-b8f7-4140-9870-5a481665636b","Type":"ContainerDied","Data":"dcf509b05e375cb0c9645d10ac1222b4981f72a0b056f08d57a8f7a286bbc092"} Sep 11 14:44:43.858605 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.858571 2578 scope.go:117] "RemoveContainer" containerID="532d1f44cfb09070e2e5ddd86f04e66f5caeb1d1887ce25532cb6eb98aeee717" Sep 11 14:44:43.868614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.868598 2578 scope.go:117] "RemoveContainer" containerID="532d1f44cfb09070e2e5ddd86f04e66f5caeb1d1887ce25532cb6eb98aeee717" Sep 11 14:44:43.868913 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:44:43.868896 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532d1f44cfb09070e2e5ddd86f04e66f5caeb1d1887ce25532cb6eb98aeee717\": container with ID starting with 532d1f44cfb09070e2e5ddd86f04e66f5caeb1d1887ce25532cb6eb98aeee717 not found: ID does not exist" containerID="532d1f44cfb09070e2e5ddd86f04e66f5caeb1d1887ce25532cb6eb98aeee717" Sep 11 14:44:43.868965 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.868922 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532d1f44cfb09070e2e5ddd86f04e66f5caeb1d1887ce25532cb6eb98aeee717"} err="failed to get container status \"532d1f44cfb09070e2e5ddd86f04e66f5caeb1d1887ce25532cb6eb98aeee717\": rpc error: code = NotFound desc = could not find container \"532d1f44cfb09070e2e5ddd86f04e66f5caeb1d1887ce25532cb6eb98aeee717\": container with ID starting with 532d1f44cfb09070e2e5ddd86f04e66f5caeb1d1887ce25532cb6eb98aeee717 not found: ID does not exist" Sep 11 14:44:43.888236 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.888211 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw"] Sep 11 14:44:43.891916 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:43.891895 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-455f2e-786f5dc897-hlwcw"] Sep 11 14:44:44.465218 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:44.465185 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f0985b-b8f7-4140-9870-5a481665636b" path="/var/lib/kubelet/pods/f5f0985b-b8f7-4140-9870-5a481665636b/volumes" Sep 11 14:44:49.131446 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:49.131399 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:44:51.591498 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:51.590890 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5"] Sep 11 14:44:51.591498 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:51.591126 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" containerID="cri-o://027edfc9145a20c08f5a2a2da6d1b2f578c86719186f9b241bf8ff8eb915d5ee" gracePeriod=30 Sep 11 14:44:55.362948 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:55.362911 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:44:55.363420 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:55.362976 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:44:56.162135 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.162097 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jjtl7_bb207b52-67df-47fb-a509-b099fafc0120/payload-processing/0.log" Sep 11 14:44:56.867142 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.867079 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:44:56.927880 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.927846 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg"] Sep 11 14:44:56.928283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928269 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f0985b-b8f7-4140-9870-5a481665636b" containerName="maas-api" Sep 11 14:44:56.928341 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928284 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f0985b-b8f7-4140-9870-5a481665636b" containerName="maas-api" Sep 11 14:44:56.928341 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928293 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" Sep 11 14:44:56.928341 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928298 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" Sep 11 14:44:56.928341 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928323 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b940871f-8c89-477c-9755-6f3f54e096df" containerName="maas-api" Sep 11 14:44:56.928341 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928328 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b940871f-8c89-477c-9755-6f3f54e096df" containerName="maas-api" Sep 11 14:44:56.928341 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928335 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4132cbff-2d1f-4ae9-acce-900ee49a1526" containerName="maas-api" Sep 11 14:44:56.928341 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928340 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4132cbff-2d1f-4ae9-acce-900ee49a1526" containerName="maas-api" Sep 11 14:44:56.928614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928352 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" Sep 11 14:44:56.928614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928357 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" Sep 11 14:44:56.928614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928364 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" Sep 11 14:44:56.928614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928369 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" Sep 11 14:44:56.928614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928477 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f0985b-b8f7-4140-9870-5a481665636b" containerName="maas-api" Sep 11 14:44:56.928614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928495 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" Sep 11 14:44:56.928614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928504 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" Sep 11 14:44:56.928614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928520 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" Sep 11 14:44:56.928614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928527 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b940871f-8c89-477c-9755-6f3f54e096df" containerName="maas-api" Sep 11 14:44:56.928614 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.928532 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4132cbff-2d1f-4ae9-acce-900ee49a1526" containerName="maas-api" Sep 11 14:44:56.931705 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.931683 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:56.934888 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.934862 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-8f7967-openshift-default-dockercfg-lljwl\"" Sep 11 14:44:56.939657 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.939627 2578 generic.go:356] "Generic (PLEG): container finished" podID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerID="027edfc9145a20c08f5a2a2da6d1b2f578c86719186f9b241bf8ff8eb915d5ee" exitCode=0 Sep 11 14:44:56.939784 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.939665 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" Sep 11 14:44:56.939784 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.939704 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" event={"ID":"e7001d7a-af65-4b81-9c1f-4b31a60a150a","Type":"ContainerDied","Data":"027edfc9145a20c08f5a2a2da6d1b2f578c86719186f9b241bf8ff8eb915d5ee"} Sep 11 14:44:56.939869 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.939797 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5" event={"ID":"e7001d7a-af65-4b81-9c1f-4b31a60a150a","Type":"ContainerDied","Data":"f5621bb23baafdf6931145d254eb4091fc6946d2136c31acf389d90a49467f48"} Sep 11 14:44:56.939869 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.939812 2578 scope.go:117] "RemoveContainer" containerID="027edfc9145a20c08f5a2a2da6d1b2f578c86719186f9b241bf8ff8eb915d5ee" Sep 11 14:44:56.948955 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.948939 2578 scope.go:117] "RemoveContainer" containerID="4ba045327d5f7dc1b214e0fc036113a47ab9758da9d1c2c5245ec6122adacb6d" Sep 11 14:44:56.957846 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.957809 2578 scope.go:117] "RemoveContainer" containerID="027edfc9145a20c08f5a2a2da6d1b2f578c86719186f9b241bf8ff8eb915d5ee" Sep 11 14:44:56.958191 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:44:56.958172 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027edfc9145a20c08f5a2a2da6d1b2f578c86719186f9b241bf8ff8eb915d5ee\": container with ID starting with 027edfc9145a20c08f5a2a2da6d1b2f578c86719186f9b241bf8ff8eb915d5ee not found: ID does not exist" containerID="027edfc9145a20c08f5a2a2da6d1b2f578c86719186f9b241bf8ff8eb915d5ee" Sep 11 14:44:56.958263 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.958201 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027edfc9145a20c08f5a2a2da6d1b2f578c86719186f9b241bf8ff8eb915d5ee"} err="failed to get container status \"027edfc9145a20c08f5a2a2da6d1b2f578c86719186f9b241bf8ff8eb915d5ee\": rpc error: code = NotFound desc = could not find container \"027edfc9145a20c08f5a2a2da6d1b2f578c86719186f9b241bf8ff8eb915d5ee\": container with ID starting with 027edfc9145a20c08f5a2a2da6d1b2f578c86719186f9b241bf8ff8eb915d5ee not found: ID does not exist" Sep 11 14:44:56.958263 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.958219 2578 scope.go:117] "RemoveContainer" containerID="4ba045327d5f7dc1b214e0fc036113a47ab9758da9d1c2c5245ec6122adacb6d" Sep 11 14:44:56.958598 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:44:56.958571 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba045327d5f7dc1b214e0fc036113a47ab9758da9d1c2c5245ec6122adacb6d\": container with ID starting with 4ba045327d5f7dc1b214e0fc036113a47ab9758da9d1c2c5245ec6122adacb6d not found: ID does not exist" containerID="4ba045327d5f7dc1b214e0fc036113a47ab9758da9d1c2c5245ec6122adacb6d" Sep 11 14:44:56.958717 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.958604 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba045327d5f7dc1b214e0fc036113a47ab9758da9d1c2c5245ec6122adacb6d"} err="failed to get container status \"4ba045327d5f7dc1b214e0fc036113a47ab9758da9d1c2c5245ec6122adacb6d\": rpc error: code = NotFound desc = could not find container \"4ba045327d5f7dc1b214e0fc036113a47ab9758da9d1c2c5245ec6122adacb6d\": container with ID starting with 4ba045327d5f7dc1b214e0fc036113a47ab9758da9d1c2c5245ec6122adacb6d not found: ID does not exist" Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.962839 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-workload-socket\") pod \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.962934 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-workload-certs\") pod \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.962963 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t86zw\" (UniqueName: \"kubernetes.io/projected/e7001d7a-af65-4b81-9c1f-4b31a60a150a-kube-api-access-t86zw\") pod \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.962992 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-credential-socket\") pod \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.963025 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-podinfo\") pod \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.963058 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-data\") pod \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.963118 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-envoy\") pod \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.963146 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-token\") pod \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.963179 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istiod-ca-cert\") pod \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\" (UID: \"e7001d7a-af65-4b81-9c1f-4b31a60a150a\") " Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.963502 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e7001d7a-af65-4b81-9c1f-4b31a60a150a" (UID: "e7001d7a-af65-4b81-9c1f-4b31a60a150a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.963678 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-data" (OuterVolumeSpecName: "istio-data") pod "e7001d7a-af65-4b81-9c1f-4b31a60a150a" (UID: "e7001d7a-af65-4b81-9c1f-4b31a60a150a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.963773 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e7001d7a-af65-4b81-9c1f-4b31a60a150a" (UID: "e7001d7a-af65-4b81-9c1f-4b31a60a150a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.963866 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e7001d7a-af65-4b81-9c1f-4b31a60a150a" (UID: "e7001d7a-af65-4b81-9c1f-4b31a60a150a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.964139 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e7001d7a-af65-4b81-9c1f-4b31a60a150a" (UID: "e7001d7a-af65-4b81-9c1f-4b31a60a150a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.964620 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg"] Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.966193 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e7001d7a-af65-4b81-9c1f-4b31a60a150a" (UID: "e7001d7a-af65-4b81-9c1f-4b31a60a150a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:44:56.967016 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.966532 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-token" (OuterVolumeSpecName: "istio-token") pod "e7001d7a-af65-4b81-9c1f-4b31a60a150a" (UID: "e7001d7a-af65-4b81-9c1f-4b31a60a150a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:56.968026 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.966617 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7001d7a-af65-4b81-9c1f-4b31a60a150a-kube-api-access-t86zw" (OuterVolumeSpecName: "kube-api-access-t86zw") pod "e7001d7a-af65-4b81-9c1f-4b31a60a150a" (UID: "e7001d7a-af65-4b81-9c1f-4b31a60a150a"). InnerVolumeSpecName "kube-api-access-t86zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:56.968026 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:56.966662 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e7001d7a-af65-4b81-9c1f-4b31a60a150a" (UID: "e7001d7a-af65-4b81-9c1f-4b31a60a150a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:57.064930 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.064891 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxggd\" (UniqueName: \"kubernetes.io/projected/017d392a-37a9-4498-aa27-f7bb7c0e4163-kube-api-access-sxggd\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.065102 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.064943 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-workload-certs\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.065102 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.064971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-podinfo\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.065102 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.064992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-data\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.065102 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-workload-socket\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.065102 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065089 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-token\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.065283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-envoy\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.065283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065137 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017d392a-37a9-4498-aa27-f7bb7c0e4163-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.065283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065166 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-credential-socket\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.065283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065234 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-workload-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:57.065283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065245 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-workload-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:57.065283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065254 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t86zw\" (UniqueName: \"kubernetes.io/projected/e7001d7a-af65-4b81-9c1f-4b31a60a150a-kube-api-access-t86zw\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:57.065283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065263 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-credential-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:57.065283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065271 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-podinfo\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:57.065283 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065282 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-data\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:57.065601 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065295 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-envoy\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:57.065601 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065304 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istio-token\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:57.065601 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.065311 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7001d7a-af65-4b81-9c1f-4b31a60a150a-istiod-ca-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:44:57.166088 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.165996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-envoy\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.166088 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.166041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017d392a-37a9-4498-aa27-f7bb7c0e4163-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.166317 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.166162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-credential-socket\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.166317 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.166257 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxggd\" (UniqueName: \"kubernetes.io/projected/017d392a-37a9-4498-aa27-f7bb7c0e4163-kube-api-access-sxggd\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.166434 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.166318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-workload-certs\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.166434 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.166380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-podinfo\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.166434 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.166407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-data\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.166591 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.166455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-workload-socket\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.166591 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.166527 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-credential-socket\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.166591 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.166583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-token\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.166877 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.166854 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017d392a-37a9-4498-aa27-f7bb7c0e4163-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.166964 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.166933 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-workload-socket\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.167013 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.166963 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-data\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.167146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.167126 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-workload-certs\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.168394 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.168377 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-envoy\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.168826 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.168808 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-podinfo\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.179049 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.179029 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxggd\" (UniqueName: \"kubernetes.io/projected/017d392a-37a9-4498-aa27-f7bb7c0e4163-kube-api-access-sxggd\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.181221 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.181199 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-token\") pod \"e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.242744 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.242703 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:57.333028 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.332990 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5"] Sep 11 14:44:57.344384 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.344355 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-455f2e-openshift-default-656b5f4d74-bvck5"] Sep 11 14:44:57.538990 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.538960 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg"] Sep 11 14:44:57.539840 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:44:57.539802 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017d392a_37a9_4498_aa27_f7bb7c0e4163.slice/crio-9f3a95e34852d19278247db84da3d733e63f3036059d59ffd6bc0aa5995d0e34 WatchSource:0}: Error finding container 9f3a95e34852d19278247db84da3d733e63f3036059d59ffd6bc0aa5995d0e34: Status 404 returned error can't find the container with id 9f3a95e34852d19278247db84da3d733e63f3036059d59ffd6bc0aa5995d0e34 Sep 11 14:44:57.542003 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.541973 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:44:57.542077 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.542039 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:44:57.542077 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.542069 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:44:57.947183 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.947147 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" event={"ID":"017d392a-37a9-4498-aa27-f7bb7c0e4163","Type":"ContainerStarted","Data":"4d937d98f2b664f61fb055aba26968aebef3ae528e40496ca4bc6168f8a768a2"} Sep 11 14:44:57.947615 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.947243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" event={"ID":"017d392a-37a9-4498-aa27-f7bb7c0e4163","Type":"ContainerStarted","Data":"9f3a95e34852d19278247db84da3d733e63f3036059d59ffd6bc0aa5995d0e34"} Sep 11 14:44:57.982552 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:57.982480 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" podStartSLOduration=1.982463563 podStartE2EDuration="1.982463563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:44:57.981892009 +0000 UTC m=+1786.067348368" watchObservedRunningTime="2026-09-11 14:44:57.982463563 +0000 UTC m=+1786.067919923" Sep 11 14:44:58.243724 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:58.243633 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:58.248676 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:58.248649 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:58.465512 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:58.465475 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" path="/var/lib/kubelet/pods/e7001d7a-af65-4b81-9c1f-4b31a60a150a/volumes" Sep 11 14:44:58.954959 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:58.954929 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:44:58.956317 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:44:58.956297 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:45:00.199122 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:00.199082 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88"] Sep 11 14:45:00.199707 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:00.199686 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" Sep 11 14:45:00.199850 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:00.199709 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7001d7a-af65-4b81-9c1f-4b31a60a150a" containerName="istio-proxy" Sep 11 14:45:00.199911 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:00.199850 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e553792-1de7-4dd9-b43f-d22976b0d94f" containerName="istio-proxy" Sep 11 14:45:00.204359 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:00.204338 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88" Sep 11 14:45:00.207253 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:00.207229 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\"" Sep 11 14:45:00.215743 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:00.215707 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88" Sep 11 14:45:00.225836 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:00.225810 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88"] Sep 11 14:45:00.526639 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:00.526568 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88"] Sep 11 14:45:00.530117 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:45:00.530090 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fc147b_d9e8_4999_bfd6_61123e763305.slice/crio-fa97d9e07b2035158b2a232aef3d0f64b1ccb6da92f293b25175549321bcedf0 WatchSource:0}: Error finding container fa97d9e07b2035158b2a232aef3d0f64b1ccb6da92f293b25175549321bcedf0: Status 404 returned error can't find the container with id fa97d9e07b2035158b2a232aef3d0f64b1ccb6da92f293b25175549321bcedf0 Sep 11 14:45:00.968435 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:00.968409 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88" event={"ID":"49fc147b-d9e8-4999-bfd6-61123e763305","Type":"ContainerStarted","Data":"caf076ae52a5636b1c9d89501904a66d446b508667995fa1db144bf1a91eff92"} Sep 11 14:45:00.968557 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:00.968460 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88" event={"ID":"49fc147b-d9e8-4999-bfd6-61123e763305","Type":"ContainerStarted","Data":"fa97d9e07b2035158b2a232aef3d0f64b1ccb6da92f293b25175549321bcedf0"} Sep 11 14:45:00.995620 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:00.995567 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88" podStartSLOduration=0.99555442 podStartE2EDuration="995.55442ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:00.993464337 +0000 UTC m=+1789.078920697" watchObservedRunningTime="2026-09-11 14:45:00.99555442 +0000 UTC m=+1789.081010779" Sep 11 14:45:01.976174 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:01.976143 2578 generic.go:356] "Generic (PLEG): container finished" podID="49fc147b-d9e8-4999-bfd6-61123e763305" containerID="caf076ae52a5636b1c9d89501904a66d446b508667995fa1db144bf1a91eff92" exitCode=0 Sep 11 14:45:01.976528 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:01.976224 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88" event={"ID":"49fc147b-d9e8-4999-bfd6-61123e763305","Type":"ContainerDied","Data":"caf076ae52a5636b1c9d89501904a66d446b508667995fa1db144bf1a91eff92"} Sep 11 14:45:03.114836 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:03.114811 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88" Sep 11 14:45:03.989280 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:03.989248 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88" Sep 11 14:45:03.989280 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:03.989258 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88" event={"ID":"49fc147b-d9e8-4999-bfd6-61123e763305","Type":"ContainerDied","Data":"fa97d9e07b2035158b2a232aef3d0f64b1ccb6da92f293b25175549321bcedf0"} Sep 11 14:45:03.989551 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:03.989319 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa97d9e07b2035158b2a232aef3d0f64b1ccb6da92f293b25175549321bcedf0" Sep 11 14:45:06.383047 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.382977 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86"] Sep 11 14:45:06.383410 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.383398 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49fc147b-d9e8-4999-bfd6-61123e763305" containerName="cleanup" Sep 11 14:45:06.383449 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.383412 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="49fc147b-d9e8-4999-bfd6-61123e763305" containerName="cleanup" Sep 11 14:45:06.383510 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.383501 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="49fc147b-d9e8-4999-bfd6-61123e763305" containerName="cleanup" Sep 11 14:45:06.386660 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.386644 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:06.389424 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.389401 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-8f7967-serving-cert\"" Sep 11 14:45:06.406892 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.406867 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86"] Sep 11 14:45:06.552252 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.552212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8flvl\" (UniqueName: \"kubernetes.io/projected/9fd32f15-2757-482f-9aa4-7649807b1d0b-kube-api-access-8flvl\") pod \"maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:06.552409 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.552320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:06.552409 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.552356 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd32f15-2757-482f-9aa4-7649807b1d0b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:06.552409 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.552387 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:06.653200 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.653127 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:06.653200 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.653157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd32f15-2757-482f-9aa4-7649807b1d0b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:06.653200 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.653177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:06.653472 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.653233 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flvl\" (UniqueName: \"kubernetes.io/projected/9fd32f15-2757-482f-9aa4-7649807b1d0b-kube-api-access-8flvl\") pod \"maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:06.653472 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:45:06.653268 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-serving-cert: secret "maas-api-e2e-ipp-cleanup-8f7967-serving-cert" not found Sep 11 14:45:06.653472 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:45:06.653338 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-maas-api-tls podName:9fd32f15-2757-482f-9aa4-7649807b1d0b nodeName:}" failed. No retries permitted until 2026-09-11 14:45:07.153321236 +0000 UTC m=+1795.238777580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" (UID: "9fd32f15-2757-482f-9aa4-7649807b1d0b") : secret "maas-api-e2e-ipp-cleanup-8f7967-serving-cert" not found Sep 11 14:45:06.653885 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.653863 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd32f15-2757-482f-9aa4-7649807b1d0b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:06.655636 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.655617 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:06.667038 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:06.667017 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flvl\" (UniqueName: \"kubernetes.io/projected/9fd32f15-2757-482f-9aa4-7649807b1d0b-kube-api-access-8flvl\") pod \"maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:07.157366 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:07.157325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:07.159908 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:07.159883 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:07.297293 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:07.297259 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:07.606288 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:07.606263 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86"] Sep 11 14:45:07.607694 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:45:07.607671 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd32f15_2757_482f_9aa4_7649807b1d0b.slice/crio-c469e0a53339a11b6588a25db9acd21995ce5b916bac98490bbf98e185d66bab WatchSource:0}: Error finding container c469e0a53339a11b6588a25db9acd21995ce5b916bac98490bbf98e185d66bab: Status 404 returned error can't find the container with id c469e0a53339a11b6588a25db9acd21995ce5b916bac98490bbf98e185d66bab Sep 11 14:45:08.015021 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:08.014940 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" event={"ID":"9fd32f15-2757-482f-9aa4-7649807b1d0b","Type":"ContainerStarted","Data":"c469e0a53339a11b6588a25db9acd21995ce5b916bac98490bbf98e185d66bab"} Sep 11 14:45:09.022913 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:09.022879 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" event={"ID":"9fd32f15-2757-482f-9aa4-7649807b1d0b","Type":"ContainerStarted","Data":"f933eb0c13d8953fdeb4ae610bacb54825215eb043d9b16da9f37007c9a841d9"} Sep 11 14:45:09.023312 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:09.022956 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:09.054557 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:09.054498 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" podStartSLOduration=2.57690812 podStartE2EDuration="3.054481338s" totalImagesPullingTime="477.573218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:09.051348816 +0000 UTC m=+1797.136805175" watchObservedRunningTime="2026-09-11 14:45:09.054481338 +0000 UTC m=+1797.139937697" Sep 11 14:45:11.209862 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:11.209826 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:45:12.466391 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:12.466363 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:45:12.470273 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:12.470247 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:45:15.035194 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:15.035166 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:22.117013 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:22.116980 2578 generic.go:356] "Generic (PLEG): container finished" podID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerID="4d937d98f2b664f61fb055aba26968aebef3ae528e40496ca4bc6168f8a768a2" exitCode=0 Sep 11 14:45:22.117442 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:22.117054 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" event={"ID":"017d392a-37a9-4498-aa27-f7bb7c0e4163","Type":"ContainerDied","Data":"4d937d98f2b664f61fb055aba26968aebef3ae528e40496ca4bc6168f8a768a2"} Sep 11 14:45:22.117487 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:22.117439 2578 scope.go:117] "RemoveContainer" containerID="4d937d98f2b664f61fb055aba26968aebef3ae528e40496ca4bc6168f8a768a2" Sep 11 14:45:22.118542 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:22.118511 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:45:22.118660 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:22.118593 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:45:22.118660 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:22.118642 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:45:23.125288 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:23.125255 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" event={"ID":"017d392a-37a9-4498-aa27-f7bb7c0e4163","Type":"ContainerStarted","Data":"ec338d5ff2b39aded40c4271c26e5690dd0e4d3453a691107a3e1ec16179674d"} Sep 11 14:45:23.243272 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:23.243242 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:45:23.243471 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:23.243446 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 11 14:45:23.243555 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:23.243502 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" podUID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 11 14:45:24.243224 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:24.243189 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 11 14:45:24.243615 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:24.243239 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" podUID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 11 14:45:25.243693 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:25.243662 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 11 14:45:25.244083 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:25.243715 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" podUID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 11 14:45:26.243841 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:26.243806 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 11 14:45:26.244360 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:26.243869 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" podUID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 11 14:45:27.243987 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:27.243950 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 11 14:45:27.244428 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:27.244004 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" podUID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 11 14:45:28.249266 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:28.249230 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:45:28.249750 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:28.249635 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:45:28.250562 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:28.250540 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:45:42.933694 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:42.933662 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86"] Sep 11 14:45:42.934066 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:42.933907 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" podUID="9fd32f15-2757-482f-9aa4-7649807b1d0b" containerName="maas-api" containerID="cri-o://f933eb0c13d8953fdeb4ae610bacb54825215eb043d9b16da9f37007c9a841d9" gracePeriod=30 Sep 11 14:45:43.212310 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.212286 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:43.244484 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.244452 2578 generic.go:356] "Generic (PLEG): container finished" podID="9fd32f15-2757-482f-9aa4-7649807b1d0b" containerID="f933eb0c13d8953fdeb4ae610bacb54825215eb043d9b16da9f37007c9a841d9" exitCode=0 Sep 11 14:45:43.244622 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.244500 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" Sep 11 14:45:43.244622 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.244530 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" event={"ID":"9fd32f15-2757-482f-9aa4-7649807b1d0b","Type":"ContainerDied","Data":"f933eb0c13d8953fdeb4ae610bacb54825215eb043d9b16da9f37007c9a841d9"} Sep 11 14:45:43.244622 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.244576 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86" event={"ID":"9fd32f15-2757-482f-9aa4-7649807b1d0b","Type":"ContainerDied","Data":"c469e0a53339a11b6588a25db9acd21995ce5b916bac98490bbf98e185d66bab"} Sep 11 14:45:43.244622 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.244599 2578 scope.go:117] "RemoveContainer" containerID="f933eb0c13d8953fdeb4ae610bacb54825215eb043d9b16da9f37007c9a841d9" Sep 11 14:45:43.255711 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.255693 2578 scope.go:117] "RemoveContainer" containerID="f933eb0c13d8953fdeb4ae610bacb54825215eb043d9b16da9f37007c9a841d9" Sep 11 14:45:43.256028 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:45:43.256001 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f933eb0c13d8953fdeb4ae610bacb54825215eb043d9b16da9f37007c9a841d9\": container with ID starting with f933eb0c13d8953fdeb4ae610bacb54825215eb043d9b16da9f37007c9a841d9 not found: ID does not exist" containerID="f933eb0c13d8953fdeb4ae610bacb54825215eb043d9b16da9f37007c9a841d9" Sep 11 14:45:43.256098 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.256042 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f933eb0c13d8953fdeb4ae610bacb54825215eb043d9b16da9f37007c9a841d9"} err="failed to get container status \"f933eb0c13d8953fdeb4ae610bacb54825215eb043d9b16da9f37007c9a841d9\": rpc error: code = NotFound desc = could not find container \"f933eb0c13d8953fdeb4ae610bacb54825215eb043d9b16da9f37007c9a841d9\": container with ID starting with f933eb0c13d8953fdeb4ae610bacb54825215eb043d9b16da9f37007c9a841d9 not found: ID does not exist" Sep 11 14:45:43.304827 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.304800 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-maas-api-tls\") pod \"9fd32f15-2757-482f-9aa4-7649807b1d0b\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " Sep 11 14:45:43.304983 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.304880 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8flvl\" (UniqueName: \"kubernetes.io/projected/9fd32f15-2757-482f-9aa4-7649807b1d0b-kube-api-access-8flvl\") pod \"9fd32f15-2757-482f-9aa4-7649807b1d0b\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " Sep 11 14:45:43.304983 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.304974 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd32f15-2757-482f-9aa4-7649807b1d0b-openshift-service-ca\") pod \"9fd32f15-2757-482f-9aa4-7649807b1d0b\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " Sep 11 14:45:43.305107 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.305020 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-metrics-certs\") pod \"9fd32f15-2757-482f-9aa4-7649807b1d0b\" (UID: \"9fd32f15-2757-482f-9aa4-7649807b1d0b\") " Sep 11 14:45:43.305387 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.305355 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd32f15-2757-482f-9aa4-7649807b1d0b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9fd32f15-2757-482f-9aa4-7649807b1d0b" (UID: "9fd32f15-2757-482f-9aa4-7649807b1d0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:45:43.306959 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.306935 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd32f15-2757-482f-9aa4-7649807b1d0b-kube-api-access-8flvl" (OuterVolumeSpecName: "kube-api-access-8flvl") pod "9fd32f15-2757-482f-9aa4-7649807b1d0b" (UID: "9fd32f15-2757-482f-9aa4-7649807b1d0b"). InnerVolumeSpecName "kube-api-access-8flvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:45:43.307064 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.306966 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9fd32f15-2757-482f-9aa4-7649807b1d0b" (UID: "9fd32f15-2757-482f-9aa4-7649807b1d0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:45:43.307107 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.307058 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9fd32f15-2757-482f-9aa4-7649807b1d0b" (UID: "9fd32f15-2757-482f-9aa4-7649807b1d0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:45:43.406851 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.406819 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fd32f15-2757-482f-9aa4-7649807b1d0b-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:45:43.406851 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.406848 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:45:43.407073 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.406863 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fd32f15-2757-482f-9aa4-7649807b1d0b-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:45:43.407073 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.406871 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8flvl\" (UniqueName: \"kubernetes.io/projected/9fd32f15-2757-482f-9aa4-7649807b1d0b-kube-api-access-8flvl\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:45:43.573202 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.573165 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86"] Sep 11 14:45:43.580426 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:43.580400 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8f7967-5c76f644b7-l4x86"] Sep 11 14:45:44.471129 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:44.471095 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fd32f15-2757-482f-9aa4-7649807b1d0b" path="/var/lib/kubelet/pods/9fd32f15-2757-482f-9aa4-7649807b1d0b/volumes" Sep 11 14:45:47.167081 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:47.167041 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:45:52.224019 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:52.223982 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg"] Sep 11 14:45:52.224433 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:52.224177 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" podUID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerName="istio-proxy" containerID="cri-o://ec338d5ff2b39aded40c4271c26e5690dd0e4d3453a691107a3e1ec16179674d" gracePeriod=30 Sep 11 14:45:57.484798 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.484719 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:45:57.637845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.637811 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxggd\" (UniqueName: \"kubernetes.io/projected/017d392a-37a9-4498-aa27-f7bb7c0e4163-kube-api-access-sxggd\") pod \"017d392a-37a9-4498-aa27-f7bb7c0e4163\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " Sep 11 14:45:57.637845 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.637847 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-podinfo\") pod \"017d392a-37a9-4498-aa27-f7bb7c0e4163\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " Sep 11 14:45:57.638071 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.637907 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-workload-socket\") pod \"017d392a-37a9-4498-aa27-f7bb7c0e4163\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " Sep 11 14:45:57.638071 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.637935 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-token\") pod \"017d392a-37a9-4498-aa27-f7bb7c0e4163\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " Sep 11 14:45:57.638071 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.637955 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-envoy\") pod \"017d392a-37a9-4498-aa27-f7bb7c0e4163\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " Sep 11 14:45:57.638071 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.637988 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-workload-certs\") pod \"017d392a-37a9-4498-aa27-f7bb7c0e4163\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " Sep 11 14:45:57.638071 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.638012 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-credential-socket\") pod \"017d392a-37a9-4498-aa27-f7bb7c0e4163\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " Sep 11 14:45:57.638071 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.638063 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017d392a-37a9-4498-aa27-f7bb7c0e4163-istiod-ca-cert\") pod \"017d392a-37a9-4498-aa27-f7bb7c0e4163\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " Sep 11 14:45:57.638412 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.638093 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-data\") pod \"017d392a-37a9-4498-aa27-f7bb7c0e4163\" (UID: \"017d392a-37a9-4498-aa27-f7bb7c0e4163\") " Sep 11 14:45:57.638412 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.638340 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "017d392a-37a9-4498-aa27-f7bb7c0e4163" (UID: "017d392a-37a9-4498-aa27-f7bb7c0e4163"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:57.638519 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.638411 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "017d392a-37a9-4498-aa27-f7bb7c0e4163" (UID: "017d392a-37a9-4498-aa27-f7bb7c0e4163"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:57.638519 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.638439 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-workload-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:45:57.638596 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.638546 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-data" (OuterVolumeSpecName: "istio-data") pod "017d392a-37a9-4498-aa27-f7bb7c0e4163" (UID: "017d392a-37a9-4498-aa27-f7bb7c0e4163"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:57.638676 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.638654 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "017d392a-37a9-4498-aa27-f7bb7c0e4163" (UID: "017d392a-37a9-4498-aa27-f7bb7c0e4163"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:57.638828 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.638699 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017d392a-37a9-4498-aa27-f7bb7c0e4163-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "017d392a-37a9-4498-aa27-f7bb7c0e4163" (UID: "017d392a-37a9-4498-aa27-f7bb7c0e4163"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:45:57.640585 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.640562 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "017d392a-37a9-4498-aa27-f7bb7c0e4163" (UID: "017d392a-37a9-4498-aa27-f7bb7c0e4163"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:45:57.640672 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.640586 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017d392a-37a9-4498-aa27-f7bb7c0e4163-kube-api-access-sxggd" (OuterVolumeSpecName: "kube-api-access-sxggd") pod "017d392a-37a9-4498-aa27-f7bb7c0e4163" (UID: "017d392a-37a9-4498-aa27-f7bb7c0e4163"). InnerVolumeSpecName "kube-api-access-sxggd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:45:57.640672 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.640661 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "017d392a-37a9-4498-aa27-f7bb7c0e4163" (UID: "017d392a-37a9-4498-aa27-f7bb7c0e4163"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:57.640791 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.640773 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-token" (OuterVolumeSpecName: "istio-token") pod "017d392a-37a9-4498-aa27-f7bb7c0e4163" (UID: "017d392a-37a9-4498-aa27-f7bb7c0e4163"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:45:57.739196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.739108 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-workload-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:45:57.739196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.739146 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-token\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:45:57.739196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.739156 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-envoy\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:45:57.739196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.739164 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-credential-socket\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:45:57.739196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.739173 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/017d392a-37a9-4498-aa27-f7bb7c0e4163-istiod-ca-cert\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:45:57.739196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.739181 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-data\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:45:57.739196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.739188 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxggd\" (UniqueName: \"kubernetes.io/projected/017d392a-37a9-4498-aa27-f7bb7c0e4163-kube-api-access-sxggd\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:45:57.739196 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:57.739196 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/017d392a-37a9-4498-aa27-f7bb7c0e4163-istio-podinfo\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:45:58.345246 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:58.345206 2578 generic.go:356] "Generic (PLEG): container finished" podID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerID="ec338d5ff2b39aded40c4271c26e5690dd0e4d3453a691107a3e1ec16179674d" exitCode=0 Sep 11 14:45:58.345449 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:58.345261 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" Sep 11 14:45:58.345449 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:58.345259 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" event={"ID":"017d392a-37a9-4498-aa27-f7bb7c0e4163","Type":"ContainerDied","Data":"ec338d5ff2b39aded40c4271c26e5690dd0e4d3453a691107a3e1ec16179674d"} Sep 11 14:45:58.345449 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:58.345362 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg" event={"ID":"017d392a-37a9-4498-aa27-f7bb7c0e4163","Type":"ContainerDied","Data":"9f3a95e34852d19278247db84da3d733e63f3036059d59ffd6bc0aa5995d0e34"} Sep 11 14:45:58.345449 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:58.345399 2578 scope.go:117] "RemoveContainer" containerID="ec338d5ff2b39aded40c4271c26e5690dd0e4d3453a691107a3e1ec16179674d" Sep 11 14:45:58.357104 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:58.357083 2578 scope.go:117] "RemoveContainer" containerID="4d937d98f2b664f61fb055aba26968aebef3ae528e40496ca4bc6168f8a768a2" Sep 11 14:45:58.365884 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:58.365864 2578 scope.go:117] "RemoveContainer" containerID="ec338d5ff2b39aded40c4271c26e5690dd0e4d3453a691107a3e1ec16179674d" Sep 11 14:45:58.366160 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:45:58.366140 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec338d5ff2b39aded40c4271c26e5690dd0e4d3453a691107a3e1ec16179674d\": container with ID starting with ec338d5ff2b39aded40c4271c26e5690dd0e4d3453a691107a3e1ec16179674d not found: ID does not exist" containerID="ec338d5ff2b39aded40c4271c26e5690dd0e4d3453a691107a3e1ec16179674d" Sep 11 14:45:58.366202 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:58.366169 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec338d5ff2b39aded40c4271c26e5690dd0e4d3453a691107a3e1ec16179674d"} err="failed to get container status \"ec338d5ff2b39aded40c4271c26e5690dd0e4d3453a691107a3e1ec16179674d\": rpc error: code = NotFound desc = could not find container \"ec338d5ff2b39aded40c4271c26e5690dd0e4d3453a691107a3e1ec16179674d\": container with ID starting with ec338d5ff2b39aded40c4271c26e5690dd0e4d3453a691107a3e1ec16179674d not found: ID does not exist" Sep 11 14:45:58.366202 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:58.366188 2578 scope.go:117] "RemoveContainer" containerID="4d937d98f2b664f61fb055aba26968aebef3ae528e40496ca4bc6168f8a768a2" Sep 11 14:45:58.366448 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:45:58.366428 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d937d98f2b664f61fb055aba26968aebef3ae528e40496ca4bc6168f8a768a2\": container with ID starting with 4d937d98f2b664f61fb055aba26968aebef3ae528e40496ca4bc6168f8a768a2 not found: ID does not exist" containerID="4d937d98f2b664f61fb055aba26968aebef3ae528e40496ca4bc6168f8a768a2" Sep 11 14:45:58.366485 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:58.366455 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d937d98f2b664f61fb055aba26968aebef3ae528e40496ca4bc6168f8a768a2"} err="failed to get container status \"4d937d98f2b664f61fb055aba26968aebef3ae528e40496ca4bc6168f8a768a2\": rpc error: code = NotFound desc = could not find container \"4d937d98f2b664f61fb055aba26968aebef3ae528e40496ca4bc6168f8a768a2\": container with ID starting with 4d937d98f2b664f61fb055aba26968aebef3ae528e40496ca4bc6168f8a768a2 not found: ID does not exist" Sep 11 14:45:58.376943 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:58.376917 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg"] Sep 11 14:45:58.384413 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:58.384391 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8f7967-openshift-default-7c4d5d564b-xlxrg"] Sep 11 14:45:58.465309 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:45:58.465282 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="017d392a-37a9-4498-aa27-f7bb7c0e4163" path="/var/lib/kubelet/pods/017d392a-37a9-4498-aa27-f7bb7c0e4163/volumes" Sep 11 14:46:09.929003 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:09.928968 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:46:12.120561 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.120524 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh"] Sep 11 14:46:12.120989 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.120768 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" podUID="600ad8ea-9f04-4b4d-9206-57e9a46a202f" containerName="maas-api" containerID="cri-o://8038690ea7ae077249421c8ec8daccb2b801f36f01469bab323a57ee640e38d6" gracePeriod=30 Sep 11 14:46:12.173529 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.173499 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88"] Sep 11 14:46:12.181026 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.180991 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0ee6c-29818965-fpt88"] Sep 11 14:46:12.374881 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.374827 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:46:12.436046 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.436015 2578 generic.go:356] "Generic (PLEG): container finished" podID="600ad8ea-9f04-4b4d-9206-57e9a46a202f" containerID="8038690ea7ae077249421c8ec8daccb2b801f36f01469bab323a57ee640e38d6" exitCode=0 Sep 11 14:46:12.436185 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.436057 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" Sep 11 14:46:12.436185 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.436100 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" event={"ID":"600ad8ea-9f04-4b4d-9206-57e9a46a202f","Type":"ContainerDied","Data":"8038690ea7ae077249421c8ec8daccb2b801f36f01469bab323a57ee640e38d6"} Sep 11 14:46:12.436185 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.436149 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" event={"ID":"600ad8ea-9f04-4b4d-9206-57e9a46a202f","Type":"ContainerDied","Data":"b8347bc0f67b897212df9fdc498eedf52b79d0d097ca56c2542d5a71618d21bb"} Sep 11 14:46:12.436185 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.436171 2578 scope.go:117] "RemoveContainer" containerID="8038690ea7ae077249421c8ec8daccb2b801f36f01469bab323a57ee640e38d6" Sep 11 14:46:12.445109 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.445093 2578 scope.go:117] "RemoveContainer" containerID="8038690ea7ae077249421c8ec8daccb2b801f36f01469bab323a57ee640e38d6" Sep 11 14:46:12.445377 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:46:12.445357 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8038690ea7ae077249421c8ec8daccb2b801f36f01469bab323a57ee640e38d6\": container with ID starting with 8038690ea7ae077249421c8ec8daccb2b801f36f01469bab323a57ee640e38d6 not found: ID does not exist" containerID="8038690ea7ae077249421c8ec8daccb2b801f36f01469bab323a57ee640e38d6" Sep 11 14:46:12.445450 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.445384 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8038690ea7ae077249421c8ec8daccb2b801f36f01469bab323a57ee640e38d6"} err="failed to get container status \"8038690ea7ae077249421c8ec8daccb2b801f36f01469bab323a57ee640e38d6\": rpc error: code = NotFound desc = could not find container \"8038690ea7ae077249421c8ec8daccb2b801f36f01469bab323a57ee640e38d6\": container with ID starting with 8038690ea7ae077249421c8ec8daccb2b801f36f01469bab323a57ee640e38d6 not found: ID does not exist" Sep 11 14:46:12.458711 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.458691 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6sb4\" (UniqueName: \"kubernetes.io/projected/600ad8ea-9f04-4b4d-9206-57e9a46a202f-kube-api-access-m6sb4\") pod \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " Sep 11 14:46:12.458873 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.458743 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600ad8ea-9f04-4b4d-9206-57e9a46a202f-metrics-certs\") pod \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " Sep 11 14:46:12.458873 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.458831 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600ad8ea-9f04-4b4d-9206-57e9a46a202f-maas-api-tls\") pod \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " Sep 11 14:46:12.458873 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.458872 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600ad8ea-9f04-4b4d-9206-57e9a46a202f-openshift-service-ca\") pod \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\" (UID: \"600ad8ea-9f04-4b4d-9206-57e9a46a202f\") " Sep 11 14:46:12.459309 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.459285 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600ad8ea-9f04-4b4d-9206-57e9a46a202f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "600ad8ea-9f04-4b4d-9206-57e9a46a202f" (UID: "600ad8ea-9f04-4b4d-9206-57e9a46a202f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:46:12.461035 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.460991 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/600ad8ea-9f04-4b4d-9206-57e9a46a202f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "600ad8ea-9f04-4b4d-9206-57e9a46a202f" (UID: "600ad8ea-9f04-4b4d-9206-57e9a46a202f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:46:12.461127 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.461092 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600ad8ea-9f04-4b4d-9206-57e9a46a202f-kube-api-access-m6sb4" (OuterVolumeSpecName: "kube-api-access-m6sb4") pod "600ad8ea-9f04-4b4d-9206-57e9a46a202f" (UID: "600ad8ea-9f04-4b4d-9206-57e9a46a202f"). InnerVolumeSpecName "kube-api-access-m6sb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:46:12.461372 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.461257 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/600ad8ea-9f04-4b4d-9206-57e9a46a202f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "600ad8ea-9f04-4b4d-9206-57e9a46a202f" (UID: "600ad8ea-9f04-4b4d-9206-57e9a46a202f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:46:12.465058 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.465039 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49fc147b-d9e8-4999-bfd6-61123e763305" path="/var/lib/kubelet/pods/49fc147b-d9e8-4999-bfd6-61123e763305/volumes" Sep 11 14:46:12.560443 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.560409 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/600ad8ea-9f04-4b4d-9206-57e9a46a202f-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:46:12.560443 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.560439 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6sb4\" (UniqueName: \"kubernetes.io/projected/600ad8ea-9f04-4b4d-9206-57e9a46a202f-kube-api-access-m6sb4\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:46:12.560443 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.560449 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/600ad8ea-9f04-4b4d-9206-57e9a46a202f-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:46:12.560655 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.560457 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/600ad8ea-9f04-4b4d-9206-57e9a46a202f-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:46:12.758070 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.758040 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh"] Sep 11 14:46:12.763721 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:12.763699 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh"] Sep 11 14:46:14.463970 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:14.463943 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="600ad8ea-9f04-4b4d-9206-57e9a46a202f" path="/var/lib/kubelet/pods/600ad8ea-9f04-4b4d-9206-57e9a46a202f/volumes" Sep 11 14:46:15.351693 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:15.351646 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f0ee6c-9d745f778-5p4sh" podUID="600ad8ea-9f04-4b4d-9206-57e9a46a202f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.91:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 11 14:46:22.616360 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:46:22.616325 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:47:14.690533 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:14.690500 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq"] Sep 11 14:47:14.691063 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:14.690716 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" podUID="7ee3df4d-bd69-4159-88a1-52329a7e8f1e" containerName="maas-api" containerID="cri-o://2f88769607b7380fdbb6e22585642f730dacdea521709c5b7dd2dea9e5e5dd18" gracePeriod=30 Sep 11 14:47:14.941665 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:14.941610 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:47:14.992848 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:14.992820 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-maas-api-tls\") pod \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " Sep 11 14:47:14.993029 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:14.992907 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-openshift-service-ca\") pod \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " Sep 11 14:47:14.993029 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:14.992951 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-metrics-certs\") pod \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " Sep 11 14:47:14.993029 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:14.992973 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jvsj\" (UniqueName: \"kubernetes.io/projected/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-kube-api-access-5jvsj\") pod \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\" (UID: \"7ee3df4d-bd69-4159-88a1-52329a7e8f1e\") " Sep 11 14:47:14.993312 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:14.993289 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ee3df4d-bd69-4159-88a1-52329a7e8f1e" (UID: "7ee3df4d-bd69-4159-88a1-52329a7e8f1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:47:14.994927 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:14.994901 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ee3df4d-bd69-4159-88a1-52329a7e8f1e" (UID: "7ee3df4d-bd69-4159-88a1-52329a7e8f1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:47:14.995043 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:14.995021 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-kube-api-access-5jvsj" (OuterVolumeSpecName: "kube-api-access-5jvsj") pod "7ee3df4d-bd69-4159-88a1-52329a7e8f1e" (UID: "7ee3df4d-bd69-4159-88a1-52329a7e8f1e"). InnerVolumeSpecName "kube-api-access-5jvsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:47:14.995212 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:14.995190 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ee3df4d-bd69-4159-88a1-52329a7e8f1e" (UID: "7ee3df4d-bd69-4159-88a1-52329a7e8f1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:47:15.094629 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:15.094597 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:47:15.094629 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:15.094625 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jvsj\" (UniqueName: \"kubernetes.io/projected/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-kube-api-access-5jvsj\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:47:15.094629 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:15.094635 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:47:15.094874 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:15.094644 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee3df4d-bd69-4159-88a1-52329a7e8f1e-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:47:15.805334 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:15.805297 2578 generic.go:356] "Generic (PLEG): container finished" podID="7ee3df4d-bd69-4159-88a1-52329a7e8f1e" containerID="2f88769607b7380fdbb6e22585642f730dacdea521709c5b7dd2dea9e5e5dd18" exitCode=0 Sep 11 14:47:15.805719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:15.805349 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" Sep 11 14:47:15.805719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:15.805381 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" event={"ID":"7ee3df4d-bd69-4159-88a1-52329a7e8f1e","Type":"ContainerDied","Data":"2f88769607b7380fdbb6e22585642f730dacdea521709c5b7dd2dea9e5e5dd18"} Sep 11 14:47:15.805719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:15.805435 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq" event={"ID":"7ee3df4d-bd69-4159-88a1-52329a7e8f1e","Type":"ContainerDied","Data":"e4a245247a2a50d31d6692e5297f9ac45b7ca830072aa2e60b266568e35bd962"} Sep 11 14:47:15.805719 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:15.805457 2578 scope.go:117] "RemoveContainer" containerID="2f88769607b7380fdbb6e22585642f730dacdea521709c5b7dd2dea9e5e5dd18" Sep 11 14:47:15.814992 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:15.814972 2578 scope.go:117] "RemoveContainer" containerID="2f88769607b7380fdbb6e22585642f730dacdea521709c5b7dd2dea9e5e5dd18" Sep 11 14:47:15.815258 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:47:15.815236 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f88769607b7380fdbb6e22585642f730dacdea521709c5b7dd2dea9e5e5dd18\": container with ID starting with 2f88769607b7380fdbb6e22585642f730dacdea521709c5b7dd2dea9e5e5dd18 not found: ID does not exist" containerID="2f88769607b7380fdbb6e22585642f730dacdea521709c5b7dd2dea9e5e5dd18" Sep 11 14:47:15.815302 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:15.815266 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f88769607b7380fdbb6e22585642f730dacdea521709c5b7dd2dea9e5e5dd18"} err="failed to get container status \"2f88769607b7380fdbb6e22585642f730dacdea521709c5b7dd2dea9e5e5dd18\": rpc error: code = NotFound desc = could not find container \"2f88769607b7380fdbb6e22585642f730dacdea521709c5b7dd2dea9e5e5dd18\": container with ID starting with 2f88769607b7380fdbb6e22585642f730dacdea521709c5b7dd2dea9e5e5dd18 not found: ID does not exist" Sep 11 14:47:15.847898 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:15.847877 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq"] Sep 11 14:47:15.874509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:15.874480 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-40d3a2-9c57cc544-fh6sq"] Sep 11 14:47:16.472747 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:16.472695 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee3df4d-bd69-4159-88a1-52329a7e8f1e" path="/var/lib/kubelet/pods/7ee3df4d-bd69-4159-88a1-52329a7e8f1e/volumes" Sep 11 14:47:18.031767 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:18.031721 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:47:37.032214 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:37.032175 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:47:53.235787 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.235753 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d"] Sep 11 14:47:53.236223 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.235971 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" podUID="bd34af71-497b-4c4e-a664-8cf8c4a1e9f2" containerName="maas-api" containerID="cri-o://1e0a2e322eb7dff328eca5f8b4d49ca2363d68ff9da5e9e18398c6b4cffd4a80" gracePeriod=30 Sep 11 14:47:53.489262 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.489201 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:47:53.622668 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.622637 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcd8b\" (UniqueName: \"kubernetes.io/projected/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-kube-api-access-kcd8b\") pod \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " Sep 11 14:47:53.622860 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.622678 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-openshift-service-ca\") pod \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " Sep 11 14:47:53.622860 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.622719 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-maas-api-tls\") pod \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " Sep 11 14:47:53.622860 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.622792 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-metrics-certs\") pod \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\" (UID: \"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2\") " Sep 11 14:47:53.623128 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.623094 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bd34af71-497b-4c4e-a664-8cf8c4a1e9f2" (UID: "bd34af71-497b-4c4e-a664-8cf8c4a1e9f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:47:53.624963 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.624933 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bd34af71-497b-4c4e-a664-8cf8c4a1e9f2" (UID: "bd34af71-497b-4c4e-a664-8cf8c4a1e9f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:47:53.625088 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.625038 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bd34af71-497b-4c4e-a664-8cf8c4a1e9f2" (UID: "bd34af71-497b-4c4e-a664-8cf8c4a1e9f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:47:53.625088 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.625055 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-kube-api-access-kcd8b" (OuterVolumeSpecName: "kube-api-access-kcd8b") pod "bd34af71-497b-4c4e-a664-8cf8c4a1e9f2" (UID: "bd34af71-497b-4c4e-a664-8cf8c4a1e9f2"). InnerVolumeSpecName "kube-api-access-kcd8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:47:53.723773 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.723715 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:47:53.723773 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.723767 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcd8b\" (UniqueName: \"kubernetes.io/projected/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-kube-api-access-kcd8b\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:47:53.723773 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.723779 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:47:53.724007 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:53.723788 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:47:54.027575 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:54.027538 2578 generic.go:356] "Generic (PLEG): container finished" podID="bd34af71-497b-4c4e-a664-8cf8c4a1e9f2" containerID="1e0a2e322eb7dff328eca5f8b4d49ca2363d68ff9da5e9e18398c6b4cffd4a80" exitCode=0 Sep 11 14:47:54.027792 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:54.027588 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" Sep 11 14:47:54.027792 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:54.027623 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" event={"ID":"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2","Type":"ContainerDied","Data":"1e0a2e322eb7dff328eca5f8b4d49ca2363d68ff9da5e9e18398c6b4cffd4a80"} Sep 11 14:47:54.027792 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:54.027676 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d" event={"ID":"bd34af71-497b-4c4e-a664-8cf8c4a1e9f2","Type":"ContainerDied","Data":"c962cfda97ebf83c99fe24265c68ee3abc44400b8f3ea520a7d0f7d18fd8078f"} Sep 11 14:47:54.027792 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:54.027700 2578 scope.go:117] "RemoveContainer" containerID="1e0a2e322eb7dff328eca5f8b4d49ca2363d68ff9da5e9e18398c6b4cffd4a80" Sep 11 14:47:54.037794 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:54.037770 2578 scope.go:117] "RemoveContainer" containerID="1e0a2e322eb7dff328eca5f8b4d49ca2363d68ff9da5e9e18398c6b4cffd4a80" Sep 11 14:47:54.038065 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:47:54.038043 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e0a2e322eb7dff328eca5f8b4d49ca2363d68ff9da5e9e18398c6b4cffd4a80\": container with ID starting with 1e0a2e322eb7dff328eca5f8b4d49ca2363d68ff9da5e9e18398c6b4cffd4a80 not found: ID does not exist" containerID="1e0a2e322eb7dff328eca5f8b4d49ca2363d68ff9da5e9e18398c6b4cffd4a80" Sep 11 14:47:54.038138 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:54.038073 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0a2e322eb7dff328eca5f8b4d49ca2363d68ff9da5e9e18398c6b4cffd4a80"} err="failed to get container status \"1e0a2e322eb7dff328eca5f8b4d49ca2363d68ff9da5e9e18398c6b4cffd4a80\": rpc error: code = NotFound desc = could not find container \"1e0a2e322eb7dff328eca5f8b4d49ca2363d68ff9da5e9e18398c6b4cffd4a80\": container with ID starting with 1e0a2e322eb7dff328eca5f8b4d49ca2363d68ff9da5e9e18398c6b4cffd4a80 not found: ID does not exist" Sep 11 14:47:54.056648 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:54.056620 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d"] Sep 11 14:47:54.063152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:54.063128 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1c284e-7b6b9c879d-sdg7d"] Sep 11 14:47:54.464500 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:54.464463 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd34af71-497b-4c4e-a664-8cf8c4a1e9f2" path="/var/lib/kubelet/pods/bd34af71-497b-4c4e-a664-8cf8c4a1e9f2/volumes" Sep 11 14:47:55.928124 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:47:55.923889 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:48:05.423912 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:48:05.423830 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:49:29.035905 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:49:29.035870 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:49:56.124109 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:49:56.124022 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:50:03.894488 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:03.894457 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:50:11.225606 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:11.225575 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:50:12.509176 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:12.509144 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:50:12.518199 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:12.518177 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8k54l_909ffdc3-b00a-4ae3-b184-5381ab8e7d13/ovn-acl-logging/0.log" Sep 11 14:50:18.858623 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:18.858590 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:50:25.939274 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:25.939236 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:50:33.759723 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:33.759682 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:50:42.534071 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:42.534040 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:50:49.561721 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:49.561679 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst"] Sep 11 14:50:49.562224 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:49.561963 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" podUID="5dcb07bc-03da-405b-b826-1d8c3ceb8d2a" containerName="manager" containerID="cri-o://f0de0621202c3b6008f3ae441b66cb0c6eca3a5612f63f5bd04eb3407e714372" gracePeriod=10 Sep 11 14:50:49.814993 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:49.814939 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" Sep 11 14:50:49.929030 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:49.928999 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7wln\" (UniqueName: \"kubernetes.io/projected/5dcb07bc-03da-405b-b826-1d8c3ceb8d2a-kube-api-access-g7wln\") pod \"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a\" (UID: \"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a\") " Sep 11 14:50:49.929210 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:49.929035 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dcb07bc-03da-405b-b826-1d8c3ceb8d2a-extensions-socket-volume\") pod \"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a\" (UID: \"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a\") " Sep 11 14:50:49.929440 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:49.929416 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcb07bc-03da-405b-b826-1d8c3ceb8d2a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5dcb07bc-03da-405b-b826-1d8c3ceb8d2a" (UID: "5dcb07bc-03da-405b-b826-1d8c3ceb8d2a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:50:49.931102 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:49.931082 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcb07bc-03da-405b-b826-1d8c3ceb8d2a-kube-api-access-g7wln" (OuterVolumeSpecName: "kube-api-access-g7wln") pod "5dcb07bc-03da-405b-b826-1d8c3ceb8d2a" (UID: "5dcb07bc-03da-405b-b826-1d8c3ceb8d2a"). InnerVolumeSpecName "kube-api-access-g7wln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:50:50.030144 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:50.030113 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7wln\" (UniqueName: \"kubernetes.io/projected/5dcb07bc-03da-405b-b826-1d8c3ceb8d2a-kube-api-access-g7wln\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:50:50.030144 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:50.030145 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dcb07bc-03da-405b-b826-1d8c3ceb8d2a-extensions-socket-volume\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:50:50.034680 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:50.034655 2578 generic.go:356] "Generic (PLEG): container finished" podID="5dcb07bc-03da-405b-b826-1d8c3ceb8d2a" containerID="f0de0621202c3b6008f3ae441b66cb0c6eca3a5612f63f5bd04eb3407e714372" exitCode=0 Sep 11 14:50:50.034803 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:50.034708 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" Sep 11 14:50:50.034803 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:50.034724 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" event={"ID":"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a","Type":"ContainerDied","Data":"f0de0621202c3b6008f3ae441b66cb0c6eca3a5612f63f5bd04eb3407e714372"} Sep 11 14:50:50.034803 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:50.034801 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst" event={"ID":"5dcb07bc-03da-405b-b826-1d8c3ceb8d2a","Type":"ContainerDied","Data":"d2e2ce43188f1393c49d05ede5c9effe2a479f10f6f34f6129b5aef514f69715"} Sep 11 14:50:50.034905 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:50.034822 2578 scope.go:117] "RemoveContainer" containerID="f0de0621202c3b6008f3ae441b66cb0c6eca3a5612f63f5bd04eb3407e714372" Sep 11 14:50:50.044892 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:50.044871 2578 scope.go:117] "RemoveContainer" containerID="f0de0621202c3b6008f3ae441b66cb0c6eca3a5612f63f5bd04eb3407e714372" Sep 11 14:50:50.045159 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:50:50.045139 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0de0621202c3b6008f3ae441b66cb0c6eca3a5612f63f5bd04eb3407e714372\": container with ID starting with f0de0621202c3b6008f3ae441b66cb0c6eca3a5612f63f5bd04eb3407e714372 not found: ID does not exist" containerID="f0de0621202c3b6008f3ae441b66cb0c6eca3a5612f63f5bd04eb3407e714372" Sep 11 14:50:50.045212 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:50.045167 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0de0621202c3b6008f3ae441b66cb0c6eca3a5612f63f5bd04eb3407e714372"} err="failed to get container status \"f0de0621202c3b6008f3ae441b66cb0c6eca3a5612f63f5bd04eb3407e714372\": rpc error: code = NotFound desc = could not find container \"f0de0621202c3b6008f3ae441b66cb0c6eca3a5612f63f5bd04eb3407e714372\": container with ID starting with f0de0621202c3b6008f3ae441b66cb0c6eca3a5612f63f5bd04eb3407e714372 not found: ID does not exist" Sep 11 14:50:50.064106 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:50.064082 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst"] Sep 11 14:50:50.069906 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:50.069858 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-66qst"] Sep 11 14:50:50.464304 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:50:50.464274 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dcb07bc-03da-405b-b826-1d8c3ceb8d2a" path="/var/lib/kubelet/pods/5dcb07bc-03da-405b-b826-1d8c3ceb8d2a/volumes" Sep 11 14:51:12.763455 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:12.763377 2578 scope.go:117] "RemoveContainer" containerID="caf076ae52a5636b1c9d89501904a66d446b508667995fa1db144bf1a91eff92" Sep 11 14:51:50.312873 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.312836 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8"] Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313228 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd34af71-497b-4c4e-a664-8cf8c4a1e9f2" containerName="maas-api" Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313239 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd34af71-497b-4c4e-a664-8cf8c4a1e9f2" containerName="maas-api" Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313246 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="600ad8ea-9f04-4b4d-9206-57e9a46a202f" containerName="maas-api" Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313252 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="600ad8ea-9f04-4b4d-9206-57e9a46a202f" containerName="maas-api" Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313266 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dcb07bc-03da-405b-b826-1d8c3ceb8d2a" containerName="manager" Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313271 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dcb07bc-03da-405b-b826-1d8c3ceb8d2a" containerName="manager" Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313279 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerName="istio-proxy" Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313284 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerName="istio-proxy" Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313297 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee3df4d-bd69-4159-88a1-52329a7e8f1e" containerName="maas-api" Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313302 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee3df4d-bd69-4159-88a1-52329a7e8f1e" containerName="maas-api" Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313313 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerName="istio-proxy" Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313318 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerName="istio-proxy" Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313325 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd32f15-2757-482f-9aa4-7649807b1d0b" containerName="maas-api" Sep 11 14:51:50.313347 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313330 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd32f15-2757-482f-9aa4-7649807b1d0b" containerName="maas-api" Sep 11 14:51:50.313816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313385 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerName="istio-proxy" Sep 11 14:51:50.313816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313393 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="600ad8ea-9f04-4b4d-9206-57e9a46a202f" containerName="maas-api" Sep 11 14:51:50.313816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313399 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd34af71-497b-4c4e-a664-8cf8c4a1e9f2" containerName="maas-api" Sep 11 14:51:50.313816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313405 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dcb07bc-03da-405b-b826-1d8c3ceb8d2a" containerName="manager" Sep 11 14:51:50.313816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313411 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee3df4d-bd69-4159-88a1-52329a7e8f1e" containerName="maas-api" Sep 11 14:51:50.313816 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.313417 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fd32f15-2757-482f-9aa4-7649807b1d0b" containerName="maas-api" Sep 11 14:51:50.316548 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.316529 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" Sep 11 14:51:50.322163 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:51:50.322126 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-dvhz8\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" podUID="73c4d455-2f0a-4e96-9dd1-b86971533434" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" Sep 11 14:51:50.322163 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:51:50.322146 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kuadrant-operator-controller-manager-dockercfg-kvhdm\" is forbidden: User \"system:node:ip-10-0-131-57.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-57.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kvhdm\"" type="*v1.Secret" Sep 11 14:51:50.332981 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.332956 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8"] Sep 11 14:51:50.406920 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.406886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8lk\" (UniqueName: \"kubernetes.io/projected/73c4d455-2f0a-4e96-9dd1-b86971533434-kube-api-access-dl8lk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dvhz8\" (UID: \"73c4d455-2f0a-4e96-9dd1-b86971533434\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" Sep 11 14:51:50.407105 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.406962 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73c4d455-2f0a-4e96-9dd1-b86971533434-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dvhz8\" (UID: \"73c4d455-2f0a-4e96-9dd1-b86971533434\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" Sep 11 14:51:50.508125 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.508096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73c4d455-2f0a-4e96-9dd1-b86971533434-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dvhz8\" (UID: \"73c4d455-2f0a-4e96-9dd1-b86971533434\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" Sep 11 14:51:50.508286 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.508171 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8lk\" (UniqueName: \"kubernetes.io/projected/73c4d455-2f0a-4e96-9dd1-b86971533434-kube-api-access-dl8lk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dvhz8\" (UID: \"73c4d455-2f0a-4e96-9dd1-b86971533434\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" Sep 11 14:51:50.508463 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.508439 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73c4d455-2f0a-4e96-9dd1-b86971533434-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dvhz8\" (UID: \"73c4d455-2f0a-4e96-9dd1-b86971533434\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" Sep 11 14:51:50.521660 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:50.521635 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8lk\" (UniqueName: \"kubernetes.io/projected/73c4d455-2f0a-4e96-9dd1-b86971533434-kube-api-access-dl8lk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dvhz8\" (UID: \"73c4d455-2f0a-4e96-9dd1-b86971533434\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" Sep 11 14:51:51.627400 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:51.627372 2578 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" secret="" err="failed to sync secret cache: timed out waiting for the condition" Sep 11 14:51:51.627856 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:51.627428 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" Sep 11 14:51:51.904966 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:51.904893 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kvhdm\"" Sep 11 14:51:51.928771 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:51.921212 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8"] Sep 11 14:51:52.390040 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:52.389979 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" event={"ID":"73c4d455-2f0a-4e96-9dd1-b86971533434","Type":"ContainerStarted","Data":"568d76aa798ee657ff6c5c759a7624214997f524c403a5b6fb27cee0096fd90d"} Sep 11 14:51:52.390222 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:52.390046 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" event={"ID":"73c4d455-2f0a-4e96-9dd1-b86971533434","Type":"ContainerStarted","Data":"9825b6aa615fcfb1d5dd610aa28cbdab7ba2eef50933984124b3013ba475620b"} Sep 11 14:51:52.390222 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:52.390177 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" Sep 11 14:51:52.423376 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:51:52.423331 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" podStartSLOduration=2.423316262 podStartE2EDuration="2.423316262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:51:52.419549089 +0000 UTC m=+2200.505005460" watchObservedRunningTime="2026-09-11 14:51:52.423316262 +0000 UTC m=+2200.508772621" Sep 11 14:52:03.397984 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:52:03.397950 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dvhz8" Sep 11 14:52:10.726277 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:52:10.726243 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:52:17.460154 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:52:17.460120 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:52:26.845589 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:52:26.845550 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:52:34.251927 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:52:34.251883 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:52:46.532439 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:52:46.532357 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:53:11.838688 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:11.838653 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:53:18.519129 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:18.519098 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:53:25.742062 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:25.742028 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:53:30.665295 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.665261 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67df577d97-5mkbn"] Sep 11 14:53:30.666944 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.666905 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="017d392a-37a9-4498-aa27-f7bb7c0e4163" containerName="istio-proxy" Sep 11 14:53:30.670671 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.670650 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:30.676026 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.676005 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 14:53:30.677265 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.677245 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:53:30.677389 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.677327 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7cfns\"" Sep 11 14:53:30.677550 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.677530 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 14:53:30.678472 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.677531 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 14:53:30.680374 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.680352 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67df577d97-5mkbn"] Sep 11 14:53:30.686955 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.686933 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk"] Sep 11 14:53:30.690507 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.690490 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:30.707235 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.707215 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk"] Sep 11 14:53:30.755569 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.755539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd71f13-1a39-443c-a00d-f1b3b619392a-openshift-service-ca\") pod \"maas-api-67df577d97-5mkbn\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:30.755752 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.755604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd71f13-1a39-443c-a00d-f1b3b619392a-maas-api-tls\") pod \"maas-api-67df577d97-5mkbn\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:30.755752 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.755689 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd71f13-1a39-443c-a00d-f1b3b619392a-metrics-certs\") pod \"maas-api-67df577d97-5mkbn\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:30.755904 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.755814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5rct\" (UniqueName: \"kubernetes.io/projected/bcd71f13-1a39-443c-a00d-f1b3b619392a-kube-api-access-v5rct\") pod \"maas-api-67df577d97-5mkbn\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:30.856386 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.856357 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770f4450-dfdb-43d4-a574-63a027f84fc0-maas-api-tls\") pod \"maas-api-54b97d6fdf-42txk\" (UID: \"770f4450-dfdb-43d4-a574-63a027f84fc0\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:30.856518 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.856408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgv46\" (UniqueName: \"kubernetes.io/projected/770f4450-dfdb-43d4-a574-63a027f84fc0-kube-api-access-hgv46\") pod \"maas-api-54b97d6fdf-42txk\" (UID: \"770f4450-dfdb-43d4-a574-63a027f84fc0\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:30.856518 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.856464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5rct\" (UniqueName: \"kubernetes.io/projected/bcd71f13-1a39-443c-a00d-f1b3b619392a-kube-api-access-v5rct\") pod \"maas-api-67df577d97-5mkbn\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:30.856518 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.856511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/770f4450-dfdb-43d4-a574-63a027f84fc0-metrics-certs\") pod \"maas-api-54b97d6fdf-42txk\" (UID: \"770f4450-dfdb-43d4-a574-63a027f84fc0\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:30.856680 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.856546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd71f13-1a39-443c-a00d-f1b3b619392a-openshift-service-ca\") pod \"maas-api-67df577d97-5mkbn\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:30.856680 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.856578 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770f4450-dfdb-43d4-a574-63a027f84fc0-openshift-service-ca\") pod \"maas-api-54b97d6fdf-42txk\" (UID: \"770f4450-dfdb-43d4-a574-63a027f84fc0\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:30.856680 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.856613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd71f13-1a39-443c-a00d-f1b3b619392a-maas-api-tls\") pod \"maas-api-67df577d97-5mkbn\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:30.856680 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.856630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd71f13-1a39-443c-a00d-f1b3b619392a-metrics-certs\") pod \"maas-api-67df577d97-5mkbn\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:30.857242 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.857221 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd71f13-1a39-443c-a00d-f1b3b619392a-openshift-service-ca\") pod \"maas-api-67df577d97-5mkbn\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:30.859044 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.859025 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd71f13-1a39-443c-a00d-f1b3b619392a-maas-api-tls\") pod \"maas-api-67df577d97-5mkbn\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:30.859146 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.859056 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd71f13-1a39-443c-a00d-f1b3b619392a-metrics-certs\") pod \"maas-api-67df577d97-5mkbn\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:30.868780 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.868752 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5rct\" (UniqueName: \"kubernetes.io/projected/bcd71f13-1a39-443c-a00d-f1b3b619392a-kube-api-access-v5rct\") pod \"maas-api-67df577d97-5mkbn\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:30.935953 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.935891 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:53:30.957396 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.957366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgv46\" (UniqueName: \"kubernetes.io/projected/770f4450-dfdb-43d4-a574-63a027f84fc0-kube-api-access-hgv46\") pod \"maas-api-54b97d6fdf-42txk\" (UID: \"770f4450-dfdb-43d4-a574-63a027f84fc0\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:30.957521 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.957422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/770f4450-dfdb-43d4-a574-63a027f84fc0-metrics-certs\") pod \"maas-api-54b97d6fdf-42txk\" (UID: \"770f4450-dfdb-43d4-a574-63a027f84fc0\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:30.957684 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.957664 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770f4450-dfdb-43d4-a574-63a027f84fc0-openshift-service-ca\") pod \"maas-api-54b97d6fdf-42txk\" (UID: \"770f4450-dfdb-43d4-a574-63a027f84fc0\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:30.957809 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.957787 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770f4450-dfdb-43d4-a574-63a027f84fc0-maas-api-tls\") pod \"maas-api-54b97d6fdf-42txk\" (UID: \"770f4450-dfdb-43d4-a574-63a027f84fc0\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:30.958331 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.958308 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770f4450-dfdb-43d4-a574-63a027f84fc0-openshift-service-ca\") pod \"maas-api-54b97d6fdf-42txk\" (UID: \"770f4450-dfdb-43d4-a574-63a027f84fc0\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:30.960100 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.960069 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770f4450-dfdb-43d4-a574-63a027f84fc0-maas-api-tls\") pod \"maas-api-54b97d6fdf-42txk\" (UID: \"770f4450-dfdb-43d4-a574-63a027f84fc0\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:30.960203 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.960076 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/770f4450-dfdb-43d4-a574-63a027f84fc0-metrics-certs\") pod \"maas-api-54b97d6fdf-42txk\" (UID: \"770f4450-dfdb-43d4-a574-63a027f84fc0\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:30.968340 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.968316 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgv46\" (UniqueName: \"kubernetes.io/projected/770f4450-dfdb-43d4-a574-63a027f84fc0-kube-api-access-hgv46\") pod \"maas-api-54b97d6fdf-42txk\" (UID: \"770f4450-dfdb-43d4-a574-63a027f84fc0\") " pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:30.982031 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:30.982012 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:31.002212 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:31.002189 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:31.270429 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:31.270395 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67df577d97-5mkbn"] Sep 11 14:53:31.271767 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:53:31.271723 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd71f13_1a39_443c_a00d_f1b3b619392a.slice/crio-9dec262d524ae2301a3113cad4e002d874e215966a22bf71c965fa04a0a2a9ea WatchSource:0}: Error finding container 9dec262d524ae2301a3113cad4e002d874e215966a22bf71c965fa04a0a2a9ea: Status 404 returned error can't find the container with id 9dec262d524ae2301a3113cad4e002d874e215966a22bf71c965fa04a0a2a9ea Sep 11 14:53:31.273375 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:31.273357 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:53:31.299661 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:31.299640 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk"] Sep 11 14:53:31.300387 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:53:31.300366 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770f4450_dfdb_43d4_a574_63a027f84fc0.slice/crio-cd2fc924e94fab641636f1e6f510952519448bc6c0a2234cec5a718b6c5a5ad9 WatchSource:0}: Error finding container cd2fc924e94fab641636f1e6f510952519448bc6c0a2234cec5a718b6c5a5ad9: Status 404 returned error can't find the container with id cd2fc924e94fab641636f1e6f510952519448bc6c0a2234cec5a718b6c5a5ad9 Sep 11 14:53:31.953178 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:31.953144 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" event={"ID":"bcd71f13-1a39-443c-a00d-f1b3b619392a","Type":"ContainerStarted","Data":"75e24560b34ede138bfe108196b57053d340dc587cedd1e87db0f724ee1e05d1"} Sep 11 14:53:31.953629 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:31.953206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" event={"ID":"bcd71f13-1a39-443c-a00d-f1b3b619392a","Type":"ContainerStarted","Data":"9dec262d524ae2301a3113cad4e002d874e215966a22bf71c965fa04a0a2a9ea"} Sep 11 14:53:31.953629 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:31.953416 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:31.955378 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:31.955343 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" event={"ID":"770f4450-dfdb-43d4-a574-63a027f84fc0","Type":"ContainerStarted","Data":"92542b5cf10fe4f212724c424e34cb8e8a12eba91cb2d83f0ff602232859e63e"} Sep 11 14:53:31.955509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:31.955401 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" event={"ID":"770f4450-dfdb-43d4-a574-63a027f84fc0","Type":"ContainerStarted","Data":"cd2fc924e94fab641636f1e6f510952519448bc6c0a2234cec5a718b6c5a5ad9"} Sep 11 14:53:31.955509 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:31.955430 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:31.976855 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:31.976807 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" podStartSLOduration=1.442087431 podStartE2EDuration="1.976790679s" totalImagesPullingTime="534.703248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:31.97620292 +0000 UTC m=+2300.061659281" watchObservedRunningTime="2026-09-11 14:53:31.976790679 +0000 UTC m=+2300.062247040" Sep 11 14:53:32.001122 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:32.001040 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" podStartSLOduration=1.563401982 podStartE2EDuration="2.001025036s" totalImagesPullingTime="437.623054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:53:31.99878756 +0000 UTC m=+2300.084243920" watchObservedRunningTime="2026-09-11 14:53:32.001025036 +0000 UTC m=+2300.086481396" Sep 11 14:53:37.968597 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:37.968568 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:37.969099 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:37.968978 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54b97d6fdf-42txk" Sep 11 14:53:38.052237 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.052200 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67df577d97-5mkbn"] Sep 11 14:53:38.052458 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.052408 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" podUID="bcd71f13-1a39-443c-a00d-f1b3b619392a" containerName="maas-api" containerID="cri-o://75e24560b34ede138bfe108196b57053d340dc587cedd1e87db0f724ee1e05d1" gracePeriod=30 Sep 11 14:53:38.286220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.286183 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:53:38.304485 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.304460 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:38.419003 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.418975 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd71f13-1a39-443c-a00d-f1b3b619392a-maas-api-tls\") pod \"bcd71f13-1a39-443c-a00d-f1b3b619392a\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " Sep 11 14:53:38.419152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.419010 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5rct\" (UniqueName: \"kubernetes.io/projected/bcd71f13-1a39-443c-a00d-f1b3b619392a-kube-api-access-v5rct\") pod \"bcd71f13-1a39-443c-a00d-f1b3b619392a\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " Sep 11 14:53:38.419152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.419050 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd71f13-1a39-443c-a00d-f1b3b619392a-openshift-service-ca\") pod \"bcd71f13-1a39-443c-a00d-f1b3b619392a\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " Sep 11 14:53:38.419152 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.419097 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd71f13-1a39-443c-a00d-f1b3b619392a-metrics-certs\") pod \"bcd71f13-1a39-443c-a00d-f1b3b619392a\" (UID: \"bcd71f13-1a39-443c-a00d-f1b3b619392a\") " Sep 11 14:53:38.419467 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.419438 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd71f13-1a39-443c-a00d-f1b3b619392a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bcd71f13-1a39-443c-a00d-f1b3b619392a" (UID: "bcd71f13-1a39-443c-a00d-f1b3b619392a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:53:38.421120 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.421095 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd71f13-1a39-443c-a00d-f1b3b619392a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bcd71f13-1a39-443c-a00d-f1b3b619392a" (UID: "bcd71f13-1a39-443c-a00d-f1b3b619392a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:53:38.421255 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.421231 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd71f13-1a39-443c-a00d-f1b3b619392a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bcd71f13-1a39-443c-a00d-f1b3b619392a" (UID: "bcd71f13-1a39-443c-a00d-f1b3b619392a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:53:38.421349 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.421335 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd71f13-1a39-443c-a00d-f1b3b619392a-kube-api-access-v5rct" (OuterVolumeSpecName: "kube-api-access-v5rct") pod "bcd71f13-1a39-443c-a00d-f1b3b619392a" (UID: "bcd71f13-1a39-443c-a00d-f1b3b619392a"). InnerVolumeSpecName "kube-api-access-v5rct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:53:38.520115 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.520087 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd71f13-1a39-443c-a00d-f1b3b619392a-maas-api-tls\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:53:38.520115 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.520110 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5rct\" (UniqueName: \"kubernetes.io/projected/bcd71f13-1a39-443c-a00d-f1b3b619392a-kube-api-access-v5rct\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:53:38.520270 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.520120 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd71f13-1a39-443c-a00d-f1b3b619392a-openshift-service-ca\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:53:38.520270 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.520129 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd71f13-1a39-443c-a00d-f1b3b619392a-metrics-certs\") on node \"ip-10-0-131-57.ec2.internal\" DevicePath \"\"" Sep 11 14:53:38.997808 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.997772 2578 generic.go:356] "Generic (PLEG): container finished" podID="bcd71f13-1a39-443c-a00d-f1b3b619392a" containerID="75e24560b34ede138bfe108196b57053d340dc587cedd1e87db0f724ee1e05d1" exitCode=0 Sep 11 14:53:38.998201 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.997820 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" Sep 11 14:53:38.998201 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.997854 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" event={"ID":"bcd71f13-1a39-443c-a00d-f1b3b619392a","Type":"ContainerDied","Data":"75e24560b34ede138bfe108196b57053d340dc587cedd1e87db0f724ee1e05d1"} Sep 11 14:53:38.998201 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.997909 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67df577d97-5mkbn" event={"ID":"bcd71f13-1a39-443c-a00d-f1b3b619392a","Type":"ContainerDied","Data":"9dec262d524ae2301a3113cad4e002d874e215966a22bf71c965fa04a0a2a9ea"} Sep 11 14:53:38.998201 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:38.997932 2578 scope.go:117] "RemoveContainer" containerID="75e24560b34ede138bfe108196b57053d340dc587cedd1e87db0f724ee1e05d1" Sep 11 14:53:39.007157 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:39.007140 2578 scope.go:117] "RemoveContainer" containerID="75e24560b34ede138bfe108196b57053d340dc587cedd1e87db0f724ee1e05d1" Sep 11 14:53:39.007407 ip-10-0-131-57 kubenswrapper[2578]: E0911 14:53:39.007388 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e24560b34ede138bfe108196b57053d340dc587cedd1e87db0f724ee1e05d1\": container with ID starting with 75e24560b34ede138bfe108196b57053d340dc587cedd1e87db0f724ee1e05d1 not found: ID does not exist" containerID="75e24560b34ede138bfe108196b57053d340dc587cedd1e87db0f724ee1e05d1" Sep 11 14:53:39.007469 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:39.007415 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e24560b34ede138bfe108196b57053d340dc587cedd1e87db0f724ee1e05d1"} err="failed to get container status \"75e24560b34ede138bfe108196b57053d340dc587cedd1e87db0f724ee1e05d1\": rpc error: code = NotFound desc = could not find container \"75e24560b34ede138bfe108196b57053d340dc587cedd1e87db0f724ee1e05d1\": container with ID starting with 75e24560b34ede138bfe108196b57053d340dc587cedd1e87db0f724ee1e05d1 not found: ID does not exist" Sep 11 14:53:39.023264 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:39.023240 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67df577d97-5mkbn"] Sep 11 14:53:39.033454 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:39.029856 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-67df577d97-5mkbn"] Sep 11 14:53:40.464904 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:40.464868 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcd71f13-1a39-443c-a00d-f1b3b619392a" path="/var/lib/kubelet/pods/bcd71f13-1a39-443c-a00d-f1b3b619392a/volumes" Sep 11 14:53:45.122818 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:45.122780 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:53:50.825684 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:50.825645 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:53:57.436863 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:53:57.436824 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vw5gj"] Sep 11 14:54:00.940382 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:00.940353 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9cff8d495-dnpkm_e2aa03ae-36a5-49a7-9395-02a822a9a3d1/authorino/0.log" Sep 11 14:54:06.970344 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:06.970265 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pb8jz_c245cb01-c3b2-4e88-9958-8987f62d91a8/manager/1.log" Sep 11 14:54:07.100052 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:07.100014 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-dk4tz_ebc9d097-340d-4a73-ba10-1b716ff4b7c0/manager/1.log" Sep 11 14:54:07.771079 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:07.771046 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d8f96d84b-j8f49_911aa1fb-0413-429e-8a25-dfe6615e528c/manager/0.log" Sep 11 14:54:07.781149 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:07.781129 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d8f96d84b-j8f49_911aa1fb-0413-429e-8a25-dfe6615e528c/copy-manifests/0.log" Sep 11 14:54:08.281078 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:08.281053 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-54b97d6fdf-42txk_770f4450-dfdb-43d4-a574-63a027f84fc0/maas-api/0.log" Sep 11 14:54:08.529357 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:08.529330 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818950-jxl6h_989739d0-7c12-4d1a-b6c5-731eb4d673e4/cleanup/0.log" Sep 11 14:54:09.628768 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:09.628717 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9cff8d495-dnpkm_e2aa03ae-36a5-49a7-9395-02a822a9a3d1/authorino/0.log" Sep 11 14:54:09.765792 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:09.765765 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-9h7sk_c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb/manager/1.log" Sep 11 14:54:09.888232 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:09.888156 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-7m8nt_23e76c7f-aeae-4756-bb38-eba9de65e417/manager/1.log" Sep 11 14:54:10.011683 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:10.011655 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-96jj6_f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76/kuadrant-console-plugin/0.log" Sep 11 14:54:10.153127 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:10.153043 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-dvhz8_73c4d455-2f0a-4e96-9dd1-b86971533434/manager/0.log" Sep 11 14:54:10.294357 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:10.294334 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-vw5gj_ef9d0fa0-2947-4c16-a7f0-4b794b37c98a/limitador/0.log" Sep 11 14:54:10.429474 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:10.429398 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-5k6br_bc3863fd-b810-45a6-a97c-a5144ba4eb98/manager/0.log" Sep 11 14:54:11.187219 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:11.187183 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57f955fb8f-kkv8l_c1db64bb-87ce-4241-909f-5965e92f7a25/kube-auth-proxy/0.log" Sep 11 14:54:11.596721 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:11.596644 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jjtl7_bb207b52-67df-47fb-a509-b099fafc0120/payload-processing/0.log" Sep 11 14:54:11.723516 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:11.723485 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59b65c65d4-d5j44_c53e1dae-b5f7-468f-a5d3-17cb726fcd33/router/0.log" Sep 11 14:54:12.601028 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:12.600999 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f99mvt_e0a95587-8c5a-420e-a2bc-ac90c95a9ddc/main/0.log" Sep 11 14:54:21.770415 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:21.770387 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-crnx9_b33d7e0c-c13c-43fd-8cac-a0a5c8adc6e8/global-pull-secret-syncer/0.log" Sep 11 14:54:22.019334 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:22.019309 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vszlk_c1994440-b56e-4b0c-b414-f68944e12f56/konnectivity-agent/0.log" Sep 11 14:54:22.045220 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:22.045152 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-57.ec2.internal_ffbb18c601b83060fb6937c4ad273ac5/haproxy/0.log" Sep 11 14:54:26.662285 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:26.662255 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9cff8d495-dnpkm_e2aa03ae-36a5-49a7-9395-02a822a9a3d1/authorino/0.log" Sep 11 14:54:26.706150 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:26.706125 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-9h7sk_c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb/manager/0.log" Sep 11 14:54:26.709194 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:26.709176 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-9h7sk_c9d8e2c2-9fe3-4728-82a0-fc4ea5faaacb/manager/1.log" Sep 11 14:54:26.740996 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:26.740967 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-7m8nt_23e76c7f-aeae-4756-bb38-eba9de65e417/manager/1.log" Sep 11 14:54:26.741153 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:26.741137 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-7m8nt_23e76c7f-aeae-4756-bb38-eba9de65e417/manager/0.log" Sep 11 14:54:26.770426 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:26.770406 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-96jj6_f9b7ccb0-ec01-49ec-a1a5-abd48ce45f76/kuadrant-console-plugin/0.log" Sep 11 14:54:26.816597 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:26.816577 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-dvhz8_73c4d455-2f0a-4e96-9dd1-b86971533434/manager/0.log" Sep 11 14:54:26.844650 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:26.844634 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-vw5gj_ef9d0fa0-2947-4c16-a7f0-4b794b37c98a/limitador/0.log" Sep 11 14:54:26.910578 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:26.910561 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-5k6br_bc3863fd-b810-45a6-a97c-a5144ba4eb98/manager/0.log" Sep 11 14:54:28.322058 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.322026 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c6b33d1-1111-48ec-8b02-11d055ac3e13/alertmanager/0.log" Sep 11 14:54:28.354603 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.354579 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c6b33d1-1111-48ec-8b02-11d055ac3e13/config-reloader/0.log" Sep 11 14:54:28.383777 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.383755 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c6b33d1-1111-48ec-8b02-11d055ac3e13/kube-rbac-proxy-web/0.log" Sep 11 14:54:28.417136 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.417117 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c6b33d1-1111-48ec-8b02-11d055ac3e13/kube-rbac-proxy/0.log" Sep 11 14:54:28.447687 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.447668 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c6b33d1-1111-48ec-8b02-11d055ac3e13/kube-rbac-proxy-metric/0.log" Sep 11 14:54:28.483406 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.483385 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c6b33d1-1111-48ec-8b02-11d055ac3e13/prom-label-proxy/0.log" Sep 11 14:54:28.523890 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.523854 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c6b33d1-1111-48ec-8b02-11d055ac3e13/init-config-reloader/0.log" Sep 11 14:54:28.570466 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.570447 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-47h9n_9008fdf9-a87b-4914-8469-d60013b1f55d/cluster-monitoring-operator/0.log" Sep 11 14:54:28.620136 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.620117 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-lzz8s_4050a332-4649-4891-ac88-36a83d70b6ba/kube-state-metrics/0.log" Sep 11 14:54:28.658706 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.658673 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-lzz8s_4050a332-4649-4891-ac88-36a83d70b6ba/kube-rbac-proxy-main/0.log" Sep 11 14:54:28.698297 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.698272 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-lzz8s_4050a332-4649-4891-ac88-36a83d70b6ba/kube-rbac-proxy-self/0.log" Sep 11 14:54:28.819031 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.819007 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nz7g9_558b6beb-1341-417b-9763-85053d4d8b26/node-exporter/0.log" Sep 11 14:54:28.860658 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.860634 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nz7g9_558b6beb-1341-417b-9763-85053d4d8b26/kube-rbac-proxy/0.log" Sep 11 14:54:28.896836 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:28.896776 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nz7g9_558b6beb-1341-417b-9763-85053d4d8b26/init-textfile/0.log" Sep 11 14:54:29.137834 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:29.137806 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-h8wbz_75503659-d535-4d5b-9485-cd92d3a6cf09/kube-rbac-proxy-main/0.log" Sep 11 14:54:29.196384 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:29.196321 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-h8wbz_75503659-d535-4d5b-9485-cd92d3a6cf09/kube-rbac-proxy-self/0.log" Sep 11 14:54:29.229213 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:29.229183 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-h8wbz_75503659-d535-4d5b-9485-cd92d3a6cf09/openshift-state-metrics/0.log" Sep 11 14:54:29.531046 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:29.530973 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-s7scw_8fabcc40-b774-42df-b33e-021b8b35650d/prometheus-operator/0.log" Sep 11 14:54:29.568075 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:29.568054 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-s7scw_8fabcc40-b774-42df-b33e-021b8b35650d/kube-rbac-proxy/0.log" Sep 11 14:54:29.603992 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:29.603970 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-qld48_9584ca71-fb39-4093-9237-189e287eccc5/prometheus-operator-admission-webhook/0.log" Sep 11 14:54:29.644013 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:29.643994 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c7bbdf7f8-wtvwd_aede382b-4542-4dfc-843e-f142f3115661/telemeter-client/0.log" Sep 11 14:54:29.685821 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:29.685806 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c7bbdf7f8-wtvwd_aede382b-4542-4dfc-843e-f142f3115661/reload/0.log" Sep 11 14:54:29.718829 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:29.718812 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c7bbdf7f8-wtvwd_aede382b-4542-4dfc-843e-f142f3115661/kube-rbac-proxy/0.log" Sep 11 14:54:31.182627 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.182600 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-gp6gj_dd588e0d-e6f8-4ffc-9a47-5a5d625514fd/networking-console-plugin/0.log" Sep 11 14:54:31.670986 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.670955 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9"] Sep 11 14:54:31.671359 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.671347 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcd71f13-1a39-443c-a00d-f1b3b619392a" containerName="maas-api" Sep 11 14:54:31.671404 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.671361 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcd71f13-1a39-443c-a00d-f1b3b619392a" containerName="maas-api" Sep 11 14:54:31.671440 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.671426 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcd71f13-1a39-443c-a00d-f1b3b619392a" containerName="maas-api" Sep 11 14:54:31.674572 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.674555 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.679772 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.679707 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jc5ln\"/\"openshift-service-ca.crt\"" Sep 11 14:54:31.679932 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.679708 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jc5ln\"/\"default-dockercfg-pzsf7\"" Sep 11 14:54:31.679932 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.679755 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jc5ln\"/\"kube-root-ca.crt\"" Sep 11 14:54:31.688324 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.688294 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9"] Sep 11 14:54:31.760051 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.760022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9eb47f7e-51df-44e4-9c19-05da0b08b857-podres\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.760232 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.760081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9eb47f7e-51df-44e4-9c19-05da0b08b857-proc\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.760232 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.760151 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47plv\" (UniqueName: \"kubernetes.io/projected/9eb47f7e-51df-44e4-9c19-05da0b08b857-kube-api-access-47plv\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.760232 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.760183 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9eb47f7e-51df-44e4-9c19-05da0b08b857-sys\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.760374 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.760252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9eb47f7e-51df-44e4-9c19-05da0b08b857-lib-modules\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.861408 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.861372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9eb47f7e-51df-44e4-9c19-05da0b08b857-proc\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.861569 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.861428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47plv\" (UniqueName: \"kubernetes.io/projected/9eb47f7e-51df-44e4-9c19-05da0b08b857-kube-api-access-47plv\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.861569 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.861447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9eb47f7e-51df-44e4-9c19-05da0b08b857-sys\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.861569 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.861495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9eb47f7e-51df-44e4-9c19-05da0b08b857-lib-modules\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.861569 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.861508 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9eb47f7e-51df-44e4-9c19-05da0b08b857-proc\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.861569 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.861525 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9eb47f7e-51df-44e4-9c19-05da0b08b857-podres\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.861796 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.861589 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9eb47f7e-51df-44e4-9c19-05da0b08b857-sys\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.861796 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.861631 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9eb47f7e-51df-44e4-9c19-05da0b08b857-podres\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.861796 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.861649 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9eb47f7e-51df-44e4-9c19-05da0b08b857-lib-modules\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.871995 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.871974 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47plv\" (UniqueName: \"kubernetes.io/projected/9eb47f7e-51df-44e4-9c19-05da0b08b857-kube-api-access-47plv\") pod \"perf-node-gather-daemonset-m7sd9\" (UID: \"9eb47f7e-51df-44e4-9c19-05da0b08b857\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:31.984571 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:31.984496 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:32.262921 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:32.262894 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9"] Sep 11 14:54:32.264405 ip-10-0-131-57 kubenswrapper[2578]: W0911 14:54:32.264380 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9eb47f7e_51df_44e4_9c19_05da0b08b857.slice/crio-a2f6b87d752ddd10d3fc2d856dc4bbdf469dc297a5e0ccafec4339d4f8c03c09 WatchSource:0}: Error finding container a2f6b87d752ddd10d3fc2d856dc4bbdf469dc297a5e0ccafec4339d4f8c03c09: Status 404 returned error can't find the container with id a2f6b87d752ddd10d3fc2d856dc4bbdf469dc297a5e0ccafec4339d4f8c03c09 Sep 11 14:54:32.318846 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:32.318818 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" event={"ID":"9eb47f7e-51df-44e4-9c19-05da0b08b857","Type":"ContainerStarted","Data":"a2f6b87d752ddd10d3fc2d856dc4bbdf469dc297a5e0ccafec4339d4f8c03c09"} Sep 11 14:54:32.587078 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:32.586997 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-pkjx9_859033b0-6de0-484b-b73c-eaea2affa856/download-server/0.log" Sep 11 14:54:33.326762 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:33.326707 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" event={"ID":"9eb47f7e-51df-44e4-9c19-05da0b08b857","Type":"ContainerStarted","Data":"a86bb7750c9daa0bf7e156738315b1e93145f8dbd8964b4b458b8e7dc9451958"} Sep 11 14:54:33.327168 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:33.326925 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:33.347506 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:33.347462 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" podStartSLOduration=2.347448754 podStartE2EDuration="2.347448754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:33.347060996 +0000 UTC m=+2361.432517357" watchObservedRunningTime="2026-09-11 14:54:33.347448754 +0000 UTC m=+2361.432905114" Sep 11 14:54:34.779506 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:34.779461 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qdqt7_e9ae826c-e270-4612-9d92-9982ce13b392/dns/0.log" Sep 11 14:54:34.829440 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:34.829410 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qdqt7_e9ae826c-e270-4612-9d92-9982ce13b392/kube-rbac-proxy/0.log" Sep 11 14:54:34.967116 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:34.967084 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2fj2j_a062adf8-6207-40d7-8eb0-302fede4913a/dns-node-resolver/0.log" Sep 11 14:54:35.748723 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:35.748695 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7plbw_d4dec9ea-724d-422e-8783-4795fe390f80/node-ca/0.log" Sep 11 14:54:37.388676 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:37.388642 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57f955fb8f-kkv8l_c1db64bb-87ce-4241-909f-5965e92f7a25/kube-auth-proxy/0.log" Sep 11 14:54:37.525348 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:37.525321 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jjtl7_bb207b52-67df-47fb-a509-b099fafc0120/payload-processing/0.log" Sep 11 14:54:37.562926 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:37.562900 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59b65c65d4-d5j44_c53e1dae-b5f7-468f-a5d3-17cb726fcd33/router/0.log" Sep 11 14:54:38.581854 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:38.581819 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r5jzm_a30eb9bb-451d-4399-8180-10739866c4a4/serve-healthcheck-canary/0.log" Sep 11 14:54:39.135929 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:39.135894 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ddrcm_b7958f39-de27-45ff-a3a3-cdc8f7908526/kube-rbac-proxy/0.log" Sep 11 14:54:39.166078 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:39.166045 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ddrcm_b7958f39-de27-45ff-a3a3-cdc8f7908526/exporter/0.log" Sep 11 14:54:39.194446 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:39.194423 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ddrcm_b7958f39-de27-45ff-a3a3-cdc8f7908526/extractor/0.log" Sep 11 14:54:39.342142 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:39.342119 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-m7sd9" Sep 11 14:54:41.750869 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:41.750838 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pb8jz_c245cb01-c3b2-4e88-9958-8987f62d91a8/manager/1.log" Sep 11 14:54:41.750869 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:41.750868 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pb8jz_c245cb01-c3b2-4e88-9958-8987f62d91a8/manager/0.log" Sep 11 14:54:41.783611 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:41.783590 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-dk4tz_ebc9d097-340d-4a73-ba10-1b716ff4b7c0/manager/0.log" Sep 11 14:54:41.935404 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:41.935372 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-dk4tz_ebc9d097-340d-4a73-ba10-1b716ff4b7c0/manager/1.log" Sep 11 14:54:42.308155 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:42.308071 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d8f96d84b-j8f49_911aa1fb-0413-429e-8a25-dfe6615e528c/manager/0.log" Sep 11 14:54:42.336067 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:42.336042 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d8f96d84b-j8f49_911aa1fb-0413-429e-8a25-dfe6615e528c/copy-manifests/0.log" Sep 11 14:54:42.361556 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:42.361532 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57f955fb8f-kkv8l_c1db64bb-87ce-4241-909f-5965e92f7a25/kube-auth-proxy/0.log" Sep 11 14:54:42.468165 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:42.468132 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jjtl7_bb207b52-67df-47fb-a509-b099fafc0120/payload-processing/0.log" Sep 11 14:54:42.512874 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:42.512852 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59b65c65d4-d5j44_c53e1dae-b5f7-468f-a5d3-17cb726fcd33/router/0.log" Sep 11 14:54:44.480425 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:44.480398 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57f955fb8f-kkv8l_c1db64bb-87ce-4241-909f-5965e92f7a25/kube-auth-proxy/0.log" Sep 11 14:54:44.606038 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:44.605998 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jjtl7_bb207b52-67df-47fb-a509-b099fafc0120/payload-processing/0.log" Sep 11 14:54:44.680652 ip-10-0-131-57 kubenswrapper[2578]: I0911 14:54:44.680627 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59b65c65d4-d5j44_c53e1dae-b5f7-468f-a5d3-17cb726fcd33/router/0.log"