Sep 10 11:49:45.158423 ip-10-0-136-197 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 11:49:45.158438 ip-10-0-136-197 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 11:49:45.158448 ip-10-0-136-197 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 11:49:45.158783 ip-10-0-136-197 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 11:49:55.291621 ip-10-0-136-197 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 10 11:49:55.291636 ip-10-0-136-197 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b2b07ca9a78a4d7191f4fd1cf37a5355 -- Sep 10 11:51:32.012418 ip-10-0-136-197 systemd[1]: Starting Kubernetes Kubelet... Sep 10 11:51:32.418450 ip-10-0-136-197 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 11:51:32.418450 ip-10-0-136-197 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 11:51:32.418450 ip-10-0-136-197 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 11:51:32.418450 ip-10-0-136-197 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 11:51:32.419040 ip-10-0-136-197 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 11:51:32.419552 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.419474 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 11:51:32.423514 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423498 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 11:51:32.423550 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423516 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 11:51:32.423550 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423521 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 11:51:32.423550 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423524 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 11:51:32.423550 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423529 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 11:51:32.423550 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423532 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 11:51:32.423550 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423536 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 11:51:32.423550 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423539 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 10 11:51:32.423550 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423542 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 11:51:32.423550 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423545 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 11:51:32.423550 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423548 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 11:51:32.423550 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423550 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 11:51:32.423550 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423553 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 11:51:32.423550 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423556 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423567 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423570 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423572 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423576 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423581 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423584 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423586 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423589 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423592 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423595 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423597 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423600 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423602 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423605 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423607 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423609 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423612 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423614 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 11:51:32.423869 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423618 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423621 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423624 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423627 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423629 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423632 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423634 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423637 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423639 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423641 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423644 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423646 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423648 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423651 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423654 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423657 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423659 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423662 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423664 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423667 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 11:51:32.424351 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423669 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423672 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423674 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423677 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423679 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423682 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423684 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423686 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423689 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423691 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423694 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423696 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423698 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423701 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423704 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423706 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423709 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423711 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423714 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423717 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 11:51:32.424860 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423719 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423722 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423724 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423727 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423729 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423732 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423734 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423737 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423739 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423744 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423747 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423750 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423752 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423755 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423758 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423760 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423762 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423765 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423768 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 11:51:32.425371 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423771 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423774 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423776 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423779 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423782 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423784 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423787 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.423790 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424522 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424529 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424533 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424536 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424539 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424543 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424546 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424549 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424552 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424555 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424558 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 11:51:32.425871 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424560 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424563 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424566 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424569 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424571 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424574 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424577 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424580 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424582 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424585 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424587 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424590 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424593 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424595 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424598 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424600 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424603 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424605 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424608 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424612 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 11:51:32.426338 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424615 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424618 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424620 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424623 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424626 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424628 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424631 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424635 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424637 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424640 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424642 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424644 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424647 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424650 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424652 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424655 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424657 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424659 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424662 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424664 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 11:51:32.426842 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424667 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424669 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424671 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424674 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424676 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424680 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424684 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424688 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424690 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424693 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424697 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424700 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424702 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424704 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424707 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424710 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424712 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424716 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424718 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424721 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 11:51:32.427346 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424724 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424726 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424729 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424731 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424733 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424736 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424738 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424741 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424743 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424746 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424748 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424751 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424753 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424756 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424758 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424761 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424763 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424767 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424770 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424774 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 11:51:32.427825 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424777 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424780 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424783 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424787 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424791 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424794 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424796 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.424799 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425193 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425199 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425201 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425204 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425207 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425210 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425212 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425215 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425218 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425220 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425223 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 11:51:32.428319 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425225 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425230 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425233 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425236 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425240 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425242 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425245 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425248 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425251 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425253 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425256 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425258 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425261 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425264 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425267 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425270 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425272 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425275 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425277 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 11:51:32.428786 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425280 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425282 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425285 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425287 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425290 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425294 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425296 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425299 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425302 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425304 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425307 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425309 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425312 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425314 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425317 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425319 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425322 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425324 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425327 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425329 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 11:51:32.429274 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425331 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425334 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425337 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425339 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425341 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425346 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425350 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425352 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425355 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425358 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425361 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425363 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425366 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425368 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425371 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425374 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425376 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425379 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425382 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425384 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 11:51:32.429854 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425388 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425391 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425394 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425396 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425398 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425401 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425403 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425406 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425408 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425411 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425414 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425416 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425419 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425422 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425424 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425427 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425429 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425434 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425437 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425439 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425441 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 11:51:32.430354 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425444 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425446 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425450 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425453 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425456 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425459 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425461 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.425464 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425543 2582 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425592 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425600 2582 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425607 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425612 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425615 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425619 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425623 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425627 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425630 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425633 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425636 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425639 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425642 2582 flags.go:64] FLAG: --cgroup-root="" Sep 10 11:51:32.430901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425645 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425650 2582 flags.go:64] FLAG: --client-ca-file="" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425652 2582 flags.go:64] FLAG: --cloud-provider="external" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425655 2582 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425659 2582 flags.go:64] FLAG: --cluster-domain="" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425662 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425667 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425670 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425674 2582 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425678 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425681 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425684 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425687 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425690 2582 flags.go:64] FLAG: --contention-profiling="false" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425693 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425696 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425699 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425702 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425705 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425708 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425711 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425714 2582 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425717 2582 flags.go:64] FLAG: --enable-server="true" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425720 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425724 2582 flags.go:64] FLAG: --event-burst="100" Sep 10 11:51:32.431439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425727 2582 flags.go:64] FLAG: --event-qps="50" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425731 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425733 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425736 2582 flags.go:64] FLAG: --eviction-hard="" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425740 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425742 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425746 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425749 2582 flags.go:64] FLAG: --eviction-soft="" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425753 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425756 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425759 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425762 2582 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425765 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425770 2582 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425773 2582 flags.go:64] FLAG: --feature-gates="" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425777 2582 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425780 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425783 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425786 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425789 2582 flags.go:64] FLAG: --healthz-port="10248" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425792 2582 flags.go:64] FLAG: --help="false" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425795 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425798 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425801 2582 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 11:51:32.432058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425803 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425807 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425810 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425813 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425815 2582 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425818 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425821 2582 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425824 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425827 2582 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425830 2582 flags.go:64] FLAG: --kube-reserved="" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425833 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425836 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425839 2582 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425842 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425844 2582 flags.go:64] FLAG: --lock-file="" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425847 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425851 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425854 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425860 2582 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425863 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425865 2582 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425869 2582 flags.go:64] FLAG: --logging-format="text" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425872 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425876 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 11:51:32.432679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425879 2582 flags.go:64] FLAG: --manifest-url="" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425882 2582 flags.go:64] FLAG: --manifest-url-header="" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425886 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425889 2582 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425893 2582 flags.go:64] FLAG: --max-pods="110" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425896 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425898 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425902 2582 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425904 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425907 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425910 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425913 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425920 2582 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425923 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425926 2582 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425929 2582 flags.go:64] FLAG: --pod-cidr="" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425932 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425937 2582 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425940 2582 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425943 2582 flags.go:64] FLAG: --pods-per-core="0" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425946 2582 flags.go:64] FLAG: --port="10250" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425949 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425952 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-085a2c9397e9735d5" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425955 2582 flags.go:64] FLAG: --qos-reserved="" Sep 10 11:51:32.433437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425959 2582 flags.go:64] FLAG: --read-only-port="10255" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425961 2582 flags.go:64] FLAG: --register-node="true" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425964 2582 flags.go:64] FLAG: --register-with-taints="" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425968 2582 flags.go:64] FLAG: --registry-burst="10" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425971 2582 flags.go:64] FLAG: --registry-qps="5" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425974 2582 flags.go:64] FLAG: --reserved-cpus="" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425977 2582 flags.go:64] FLAG: --reserved-memory="" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425981 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425984 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425987 2582 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425989 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425992 2582 flags.go:64] FLAG: --runonce="false" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425995 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.425998 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426001 2582 flags.go:64] FLAG: --seccomp-default="false" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426004 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426007 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426010 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426012 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426015 2582 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426018 2582 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426021 2582 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426024 2582 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426026 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426029 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 11:51:32.434454 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426033 2582 flags.go:64] FLAG: --system-cgroups="" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426035 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426041 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426043 2582 flags.go:64] FLAG: --tls-cert-file="" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426046 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426049 2582 flags.go:64] FLAG: --tls-min-version="" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426052 2582 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426058 2582 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426061 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426064 2582 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426067 2582 flags.go:64] FLAG: --v="2" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426070 2582 flags.go:64] FLAG: --version="false" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426074 2582 flags.go:64] FLAG: --vmodule="" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426078 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.426082 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426214 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426218 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426221 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426224 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426226 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426229 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426231 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426234 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 11:51:32.435587 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426236 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426239 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426241 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426244 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426246 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426249 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426252 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426254 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426257 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426259 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426262 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426264 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426267 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426269 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426272 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426274 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426278 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426281 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426284 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 11:51:32.436327 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426286 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426289 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426291 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426294 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426296 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426299 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426301 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426304 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426308 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426312 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426315 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426317 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426320 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426324 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426327 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426330 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426333 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426335 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426338 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426341 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 11:51:32.436790 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426343 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426346 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426349 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426352 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426355 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426357 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426360 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426362 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426365 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426368 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426370 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426373 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426375 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426378 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426381 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426383 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426386 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426388 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426390 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 11:51:32.437633 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426393 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426395 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426398 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426400 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426402 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426405 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426407 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426410 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426412 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426414 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426417 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426419 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426422 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426424 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426426 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426429 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426431 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426434 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426437 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 11:51:32.438457 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426439 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426443 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426447 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426450 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426452 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426455 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426458 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426460 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426464 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426466 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426469 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426471 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426474 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.426477 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.427158 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 11:51:32.438938 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.435849 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.435869 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.436869 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.436889 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.436992 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437000 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437005 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437010 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437015 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437019 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437023 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437027 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437031 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437036 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437040 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437045 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437049 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437053 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437057 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437062 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 11:51:32.439545 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437066 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437070 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437074 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437078 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437083 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437087 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437091 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437095 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437099 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437104 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437108 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437112 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437119 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437155 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437161 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437166 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437171 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437175 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437180 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 11:51:32.440505 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437184 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437189 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437193 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437197 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437201 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437206 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437210 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437214 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437218 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437222 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437226 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437230 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437234 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437238 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437243 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437247 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437251 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437255 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437259 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 11:51:32.441229 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437263 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437267 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437271 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437275 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437279 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437283 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437287 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437292 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437304 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437309 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437314 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437320 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437325 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437329 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437333 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437337 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437341 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437346 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437350 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437354 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 11:51:32.441698 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437358 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437362 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437366 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437370 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437374 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437378 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437382 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437387 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437391 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437396 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437400 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437404 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437410 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437415 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437420 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437424 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437428 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437432 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437437 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 11:51:32.442260 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437442 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437446 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437458 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437463 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437467 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437471 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.437479 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437726 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437735 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437740 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437744 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437748 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437753 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437757 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437761 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 11:51:32.442706 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437767 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437774 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437778 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437781 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437785 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437790 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437794 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437798 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437802 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437809 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437814 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437819 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437824 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437828 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437832 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437836 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437840 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437845 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437850 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 11:51:32.443164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437854 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437868 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437873 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437877 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437881 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437885 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437889 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437893 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437897 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437902 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437908 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437913 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437918 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437922 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437927 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437931 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437935 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437939 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437943 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437947 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 11:51:32.443624 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437951 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437955 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437959 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437963 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437967 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437971 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437975 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437979 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437983 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437987 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437991 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437995 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.437999 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438003 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438014 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438018 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438022 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438026 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438030 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 11:51:32.444129 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438034 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438038 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438042 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438046 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438050 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438054 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438058 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438062 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438066 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438070 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438074 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438077 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438081 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438085 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438089 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438093 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438098 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438101 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438105 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438109 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 11:51:32.444695 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438113 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438117 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438121 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438125 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438130 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438134 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438138 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438159 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438171 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438175 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438179 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438183 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:32.438187 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.438194 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 11:51:32.445483 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.438367 2582 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 11:51:32.445940 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.440817 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 11:51:32.445940 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.441945 2582 server.go:1019] "Starting client certificate rotation" Sep 10 11:51:32.445940 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.442049 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 11:51:32.445940 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.442803 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 11:51:32.445940 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.445307 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 11:51:32.449038 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.449019 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 11:51:32.470732 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.470715 2582 log.go:25] "Validated CRI v1 runtime API" Sep 10 11:51:32.474024 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.474007 2582 log.go:25] "Validated CRI v1 image API" Sep 10 11:51:32.474873 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.474852 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 11:51:32.477486 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.477462 2582 fs.go:135] Filesystem UUIDs: map[11be9bb9-dc33-4866-8180-beb467b77107:/dev/nvme0n1p4 46885df9-5a0a-49ea-ba1e-5c1d4729d04b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 10 11:51:32.477576 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.477483 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 11:51:32.479687 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.479667 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 11:51:32.482538 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.482427 2582 manager.go:216] Machine: {Timestamp:2026-09-10 11:51:32.481056803 +0000 UTC m=+0.360446226 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097843 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b89c97c0c009e0def021d5563712f SystemUUID:ec2b89c9-7c0c-009e-0def-021d5563712f BootID:b2b07ca9-a78a-4d71-91f4-fd1cf37a5355 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c7:f6:a4:2c:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c7:f6:a4:2c:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:3e:42:37:fa:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 11:51:32.482538 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.482532 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 11:51:32.482656 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.482602 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 11:51:32.486103 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.486082 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 11:51:32.486263 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.486106 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-197.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 11:51:32.486975 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.486965 2582 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 11:51:32.487007 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.486977 2582 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 11:51:32.487007 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.486985 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 11:51:32.488590 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.488580 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 11:51:32.490256 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.490246 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 11:51:32.494452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.494442 2582 state_mem.go:36] "Initialized new in-memory state store" Sep 10 11:51:32.494571 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.494561 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 11:51:32.494611 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.494601 2582 kubelet.go:486] "Attempting to sync node with API server" Sep 10 11:51:32.494643 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.494626 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 11:51:32.494643 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.494642 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 11:51:32.494699 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.494656 2582 kubelet.go:398] "Adding apiserver pod source" Sep 10 11:51:32.494699 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.494668 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 11:51:32.495086 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.495072 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 11:51:32.495128 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.495100 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 11:51:32.497684 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.497664 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 11:51:32.500814 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.500799 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 11:51:32.500857 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.500832 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 11:51:32.501738 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.501726 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 11:51:32.501777 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.501746 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 11:51:32.501777 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.501752 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 11:51:32.501777 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.501757 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 11:51:32.501777 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.501766 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 11:51:32.501777 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.501771 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 11:51:32.501905 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.501783 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 11:51:32.501905 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.501789 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 11:51:32.501905 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.501801 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 11:51:32.501905 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.501806 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 11:51:32.501905 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.501814 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 11:51:32.501905 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.501819 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 11:51:32.502621 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.502610 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 11:51:32.502658 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.502624 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 11:51:32.506680 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.506660 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 11:51:32.506965 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.506949 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 11:51:32.507508 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.507486 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-197.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 11:51:32.508515 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.508504 2582 server.go:1268] "Started kubelet" Sep 10 11:51:32.508673 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.508629 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 11:51:32.508926 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.508622 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 11:51:32.508993 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.508952 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 11:51:32.509264 ip-10-0-136-197 systemd[1]: Started Kubernetes Kubelet. Sep 10 11:51:32.510687 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.510674 2582 server.go:310] "Adding debug handlers to kubelet server" Sep 10 11:51:32.512278 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.512261 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 11:51:32.513977 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.513959 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 11:51:32.518547 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.518526 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 11:51:32.518547 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.518542 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 11:51:32.518663 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.518616 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-197.ec2.internal\" not found" Sep 10 11:51:32.519226 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.519209 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 11:51:32.519315 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.519247 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 11:51:32.519315 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.519207 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 11:51:32.519583 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.519340 2582 reconstruct.go:97] "Volume reconstruction finished" Sep 10 11:51:32.519583 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.519345 2582 factory.go:55] Registering systemd factory Sep 10 11:51:32.519583 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.519364 2582 factory.go:223] Registration of the systemd container factory successfully Sep 10 11:51:32.519583 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.519349 2582 reconciler.go:29] "Reconciler: start to sync state" Sep 10 11:51:32.519583 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.518332 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-197.ec2.internal.18d3f365178b320d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-197.ec2.internal,UID:ip-10-0-136-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-197.ec2.internal,},FirstTimestamp:2026-09-10 11:51:32.508463629 +0000 UTC m=+0.387853053,LastTimestamp:2026-09-10 11:51:32.508463629 +0000 UTC m=+0.387853053,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-197.ec2.internal,}" Sep 10 11:51:32.519890 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.519875 2582 factory.go:153] Registering CRI-O factory Sep 10 11:51:32.519951 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.519893 2582 factory.go:223] Registration of the crio container factory successfully Sep 10 11:51:32.519951 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.519934 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 11:51:32.519951 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.519948 2582 factory.go:103] Registering Raw factory Sep 10 11:51:32.520096 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.519959 2582 manager.go:1196] Started watching for new ooms in manager Sep 10 11:51:32.520422 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.520405 2582 manager.go:349] Starting recovery of all containers Sep 10 11:51:32.521159 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.521116 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rpfrz" Sep 10 11:51:32.527421 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.527389 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 11:51:32.527701 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.527649 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 11:51:32.531554 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.531421 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rpfrz" Sep 10 11:51:32.532378 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.532363 2582 manager.go:354] Recovery completed Sep 10 11:51:32.537523 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.537509 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 11:51:32.539484 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.539466 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientMemory" Sep 10 11:51:32.539533 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.539500 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 11:51:32.539533 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.539513 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientPID" Sep 10 11:51:32.540020 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.540005 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 11:51:32.540020 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.540018 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 11:51:32.540163 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.540025 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 11:51:32.540163 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.540032 2582 state_mem.go:36] "Initialized new in-memory state store" Sep 10 11:51:32.540693 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.540680 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientMemory" Sep 10 11:51:32.540735 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.540705 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 11:51:32.540735 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.540715 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientPID" Sep 10 11:51:32.542045 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.542032 2582 policy_none.go:49] "None policy: Start" Sep 10 11:51:32.542082 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.542052 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 11:51:32.542082 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.542066 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 11:51:32.542237 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.542174 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-197.ec2.internal.18d3f36519648acb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-197.ec2.internal,UID:ip-10-0-136-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-197.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-197.ec2.internal,},FirstTimestamp:2026-09-10 11:51:32.539484875 +0000 UTC m=+0.418874297,LastTimestamp:2026-09-10 11:51:32.539484875 +0000 UTC m=+0.418874297,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-197.ec2.internal,}" Sep 10 11:51:32.544013 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.544001 2582 policy_none.go:47] "Start" Sep 10 11:51:32.576604 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.576590 2582 manager.go:337] "Starting Device Plugin manager" Sep 10 11:51:32.604508 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.576628 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 11:51:32.604508 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.576642 2582 server.go:87] "Starting device plugin registration server" Sep 10 11:51:32.604508 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.576950 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 11:51:32.604508 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.576965 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 11:51:32.604508 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.577062 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 11:51:32.604508 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.577170 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 11:51:32.604508 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.577178 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 11:51:32.604508 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.577624 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 11:51:32.604508 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.577678 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-197.ec2.internal\" not found" Sep 10 11:51:32.621562 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.621539 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 11:51:32.622767 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.622752 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 11:51:32.622826 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.622796 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 11:51:32.622826 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.622816 2582 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 11:51:32.622914 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.622850 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 11:51:32.625516 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.625500 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 11:51:32.677619 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.677535 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 11:51:32.678497 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.678480 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientMemory" Sep 10 11:51:32.678569 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.678511 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 11:51:32.678569 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.678523 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientPID" Sep 10 11:51:32.678569 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.678545 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.690755 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.690741 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.690826 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.690763 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-197.ec2.internal\": node \"ip-10-0-136-197.ec2.internal\" not found" Sep 10 11:51:32.704215 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.704198 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-197.ec2.internal\" not found" Sep 10 11:51:32.723596 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.723577 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-197.ec2.internal"] Sep 10 11:51:32.723663 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.723654 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 11:51:32.725054 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.725040 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientMemory" Sep 10 11:51:32.725111 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.725066 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 11:51:32.725111 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.725076 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientPID" Sep 10 11:51:32.727023 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.727011 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 11:51:32.727175 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.727161 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.727212 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.727190 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 11:51:32.727755 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.727736 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientMemory" Sep 10 11:51:32.727871 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.727762 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 11:51:32.727871 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.727773 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientPID" Sep 10 11:51:32.727871 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.727828 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientMemory" Sep 10 11:51:32.727871 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.727853 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 11:51:32.727871 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.727867 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientPID" Sep 10 11:51:32.729694 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.729675 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.729755 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.729709 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 11:51:32.730401 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.730385 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientMemory" Sep 10 11:51:32.730475 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.730410 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 11:51:32.730475 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.730420 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeHasSufficientPID" Sep 10 11:51:32.751642 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.751626 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-197.ec2.internal\" not found" node="ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.755212 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.755195 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-197.ec2.internal\" not found" node="ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.804981 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.804953 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-197.ec2.internal\" not found" Sep 10 11:51:32.821549 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.821528 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/07dafb2c54e80bc28435cbc46f61ca4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal\" (UID: \"07dafb2c54e80bc28435cbc46f61ca4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.821643 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.821592 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07dafb2c54e80bc28435cbc46f61ca4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal\" (UID: \"07dafb2c54e80bc28435cbc46f61ca4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.821643 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.821615 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03d9f1651513641d55fc001cd4afe684-config\") pod \"kube-apiserver-proxy-ip-10-0-136-197.ec2.internal\" (UID: \"03d9f1651513641d55fc001cd4afe684\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.905516 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:32.905481 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-197.ec2.internal\" not found" Sep 10 11:51:32.922299 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.922273 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/07dafb2c54e80bc28435cbc46f61ca4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal\" (UID: \"07dafb2c54e80bc28435cbc46f61ca4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.922399 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.922305 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07dafb2c54e80bc28435cbc46f61ca4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal\" (UID: \"07dafb2c54e80bc28435cbc46f61ca4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.922399 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.922331 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03d9f1651513641d55fc001cd4afe684-config\") pod \"kube-apiserver-proxy-ip-10-0-136-197.ec2.internal\" (UID: \"03d9f1651513641d55fc001cd4afe684\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.922399 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.922368 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03d9f1651513641d55fc001cd4afe684-config\") pod \"kube-apiserver-proxy-ip-10-0-136-197.ec2.internal\" (UID: \"03d9f1651513641d55fc001cd4afe684\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.922399 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.922375 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07dafb2c54e80bc28435cbc46f61ca4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal\" (UID: \"07dafb2c54e80bc28435cbc46f61ca4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal" Sep 10 11:51:32.922540 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:32.922368 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/07dafb2c54e80bc28435cbc46f61ca4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal\" (UID: \"07dafb2c54e80bc28435cbc46f61ca4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal" Sep 10 11:51:33.006110 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.006048 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-197.ec2.internal\" not found" Sep 10 11:51:33.053553 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.053536 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal" Sep 10 11:51:33.057303 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.057278 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-197.ec2.internal" Sep 10 11:51:33.106311 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.106279 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-197.ec2.internal\" not found" Sep 10 11:51:33.206788 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.206759 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-197.ec2.internal\" not found" Sep 10 11:51:33.307329 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.307267 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-197.ec2.internal\" not found" Sep 10 11:51:33.366751 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.366727 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 11:51:33.407946 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.407909 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-197.ec2.internal\" not found" Sep 10 11:51:33.441425 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.441406 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 11:51:33.441797 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.441526 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 11:51:33.441797 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.441569 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 11:51:33.455821 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.455801 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 11:51:33.495263 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.495226 2582 apiserver.go:52] "Watching apiserver" Sep 10 11:51:33.505781 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.505754 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 11:51:33.506288 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.506275 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 11:51:33.506732 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.506712 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-4xtfr","openshift-dns/node-resolver-ddt89","openshift-image-registry/node-ca-k62hz","openshift-multus/multus-additional-cni-plugins-rxnr5","openshift-network-diagnostics/network-check-target-xsq5c","openshift-network-operator/iptables-alerter-dmmg6","kube-system/global-pull-secret-syncer-lkhnc","openshift-multus/multus-55lvl","openshift-multus/network-metrics-daemon-lnhpb","openshift-ovn-kubernetes/ovnkube-node-gnz8p","kube-system/konnectivity-agent-wjzd5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv"] Sep 10 11:51:33.508976 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.508954 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.510865 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.510847 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ddt89" Sep 10 11:51:33.511551 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.511537 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p8hlq\"" Sep 10 11:51:33.511897 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.511883 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 11:51:33.511953 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.511889 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 11:51:33.512794 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.512779 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k62hz" Sep 10 11:51:33.514155 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.514125 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 11:51:33.514257 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.514244 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 11:51:33.514304 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.514275 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h8gnj\"" Sep 10 11:51:33.514789 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.514774 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.516822 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.516802 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:33.516902 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.516877 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:33.518042 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.518025 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 11:51:33.518042 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.518038 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 11:51:33.518180 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.518025 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 11:51:33.518180 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.518129 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q79lk\"" Sep 10 11:51:33.518283 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.518201 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 11:51:33.518336 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.518309 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 11:51:33.518498 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.518482 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 11:51:33.518570 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.518485 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 11:51:33.518570 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.518488 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6jz4s\"" Sep 10 11:51:33.518650 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.518606 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 11:51:33.518689 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.518677 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal" Sep 10 11:51:33.518831 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.518817 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 11:51:33.518936 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.518923 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dmmg6" Sep 10 11:51:33.521056 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.521042 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:33.521119 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.521104 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:33.522752 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.522737 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 11:51:33.523224 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.523204 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.523327 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.523278 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j9fb5\"" Sep 10 11:51:33.523327 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.523309 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 11:51:33.523441 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.523286 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 11:51:33.525355 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525336 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:33.525442 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-host\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.525442 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525385 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62e51f21-9e30-4c69-bca0-b1aca76a0a24-dbus\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:33.525442 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.525405 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:33.525442 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44bc3315-9704-4168-8d8e-869a507f3414-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.525442 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525440 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-modprobe-d\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.525686 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525455 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-sysctl-d\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.525686 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525471 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-lib-modules\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.525686 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525504 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-tmp\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.525686 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525527 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e58f844c-4b86-475d-b704-14d4e304d626-host-slash\") pod \"iptables-alerter-dmmg6\" (UID: \"e58f844c-4b86-475d-b704-14d4e304d626\") " pod="openshift-network-operator/iptables-alerter-dmmg6" Sep 10 11:51:33.525686 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525544 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt79v\" (UniqueName: \"kubernetes.io/projected/e58f844c-4b86-475d-b704-14d4e304d626-kube-api-access-tt79v\") pod \"iptables-alerter-dmmg6\" (UID: \"e58f844c-4b86-475d-b704-14d4e304d626\") " pod="openshift-network-operator/iptables-alerter-dmmg6" Sep 10 11:51:33.525686 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525566 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44bc3315-9704-4168-8d8e-869a507f3414-cni-binary-copy\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.525686 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525580 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jwqq\" (UniqueName: \"kubernetes.io/projected/44bc3315-9704-4168-8d8e-869a507f3414-kube-api-access-6jwqq\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.525686 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525594 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89fde479-2004-4ff5-a4ed-1ad6180e0dda-tmp-dir\") pod \"node-resolver-ddt89\" (UID: \"89fde479-2004-4ff5-a4ed-1ad6180e0dda\") " pod="openshift-dns/node-resolver-ddt89" Sep 10 11:51:33.525686 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525620 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44bc3315-9704-4168-8d8e-869a507f3414-system-cni-dir\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.525686 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525662 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-kubernetes\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.525686 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525683 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-sys\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525698 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn2bn\" (UniqueName: \"kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn\") pod \"network-check-target-xsq5c\" (UID: \"3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f\") " pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525711 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89fde479-2004-4ff5-a4ed-1ad6180e0dda-hosts-file\") pod \"node-resolver-ddt89\" (UID: \"89fde479-2004-4ff5-a4ed-1ad6180e0dda\") " pod="openshift-dns/node-resolver-ddt89" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525724 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-var-lib-kubelet\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525737 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds28m\" (UniqueName: \"kubernetes.io/projected/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-kube-api-access-ds28m\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525752 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g4xh\" (UniqueName: \"kubernetes.io/projected/5480668d-f15c-4051-a9d4-a2ca9fefe4d3-kube-api-access-5g4xh\") pod \"node-ca-k62hz\" (UID: \"5480668d-f15c-4051-a9d4-a2ca9fefe4d3\") " pod="openshift-image-registry/node-ca-k62hz" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525772 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44bc3315-9704-4168-8d8e-869a507f3414-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525797 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-tuned\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525817 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5480668d-f15c-4051-a9d4-a2ca9fefe4d3-host\") pod \"node-ca-k62hz\" (UID: \"5480668d-f15c-4051-a9d4-a2ca9fefe4d3\") " pod="openshift-image-registry/node-ca-k62hz" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525832 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-systemd\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525845 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-run\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525859 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e58f844c-4b86-475d-b704-14d4e304d626-iptables-alerter-script\") pod \"iptables-alerter-dmmg6\" (UID: \"e58f844c-4b86-475d-b704-14d4e304d626\") " pod="openshift-network-operator/iptables-alerter-dmmg6" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525883 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5480668d-f15c-4051-a9d4-a2ca9fefe4d3-serviceca\") pod \"node-ca-k62hz\" (UID: \"5480668d-f15c-4051-a9d4-a2ca9fefe4d3\") " pod="openshift-image-registry/node-ca-k62hz" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525906 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44bc3315-9704-4168-8d8e-869a507f3414-cnibin\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525948 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44bc3315-9704-4168-8d8e-869a507f3414-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.525972 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62e51f21-9e30-4c69-bca0-b1aca76a0a24-kubelet-config\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:33.526209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.526000 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:33.527079 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.526022 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6ccj\" (UniqueName: \"kubernetes.io/projected/89fde479-2004-4ff5-a4ed-1ad6180e0dda-kube-api-access-r6ccj\") pod \"node-resolver-ddt89\" (UID: \"89fde479-2004-4ff5-a4ed-1ad6180e0dda\") " pod="openshift-dns/node-resolver-ddt89" Sep 10 11:51:33.527079 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.526044 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-sysconfig\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.527079 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.526058 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-sysctl-conf\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.527079 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.526875 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zgznx\"" Sep 10 11:51:33.527546 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.527530 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.529250 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.529223 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 11:51:33.529758 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.529724 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wjzd5" Sep 10 11:51:33.530472 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.530411 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 11:51:33.530472 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.530445 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 11:51:33.530826 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.530808 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 11:51:33.530902 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.530844 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 11:51:33.530957 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.530942 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 11:51:33.530957 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.530811 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 11:51:33.531812 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.531756 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9mhmm\"" Sep 10 11:51:33.532410 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.532380 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 11:51:33.533789 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.533404 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9q5nc\"" Sep 10 11:51:33.534392 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.534273 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 11:51:33.534392 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.534387 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.534531 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.534457 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 11:51:33.534588 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.534537 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-197.ec2.internal" Sep 10 11:51:33.534786 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.534767 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal"] Sep 10 11:51:33.534852 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.534808 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 11:46:32 +0000 UTC" deadline="2028-06-13 02:31:25.43617625 +0000 UTC" Sep 10 11:51:33.534895 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.534856 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15398h39m51.901323587s" Sep 10 11:51:33.537915 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.537898 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87ssd\"" Sep 10 11:51:33.538263 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.538250 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 11:51:33.538350 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.538335 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 11:51:33.538591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.538579 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 11:51:33.541533 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.541518 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 11:51:33.548542 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.548525 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-197.ec2.internal"] Sep 10 11:51:33.549507 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.549494 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 11:51:33.576065 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.576036 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fwbxq" Sep 10 11:51:33.586083 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.586066 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fwbxq" Sep 10 11:51:33.619906 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.619883 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 11:51:33.626386 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:33.626490 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-host\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.626490 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626429 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62e51f21-9e30-4c69-bca0-b1aca76a0a24-dbus\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:33.626490 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626480 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-multus-conf-dir\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.626623 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626510 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhpsj\" (UniqueName: \"kubernetes.io/projected/bb366139-ec7c-4ae4-be68-93027b09d18c-kube-api-access-bhpsj\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:33.626623 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626536 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/483e138d-49da-44ef-9e55-0d50f70ffb75-konnectivity-ca\") pod \"konnectivity-agent-wjzd5\" (UID: \"483e138d-49da-44ef-9e55-0d50f70ffb75\") " pod="kube-system/konnectivity-agent-wjzd5" Sep 10 11:51:33.626623 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-run-netns\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.626623 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-var-lib-openvswitch\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.626623 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44bc3315-9704-4168-8d8e-869a507f3414-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.626833 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626642 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-sysctl-d\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.626833 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626664 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e58f844c-4b86-475d-b704-14d4e304d626-host-slash\") pod \"iptables-alerter-dmmg6\" (UID: \"e58f844c-4b86-475d-b704-14d4e304d626\") " pod="openshift-network-operator/iptables-alerter-dmmg6" Sep 10 11:51:33.626833 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626685 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-kubelet\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.626833 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626717 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-cni-bin\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.626833 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626743 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44bc3315-9704-4168-8d8e-869a507f3414-cni-binary-copy\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.626833 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626766 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jwqq\" (UniqueName: \"kubernetes.io/projected/44bc3315-9704-4168-8d8e-869a507f3414-kube-api-access-6jwqq\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.626833 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-system-cni-dir\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.626833 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626807 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-run-openvswitch\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.626833 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-run-ovn-kubernetes\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626851 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b98c0ee8-844a-4f6c-89ce-f9492e87509b-ovnkube-config\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626883 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b98c0ee8-844a-4f6c-89ce-f9492e87509b-ovn-node-metrics-cert\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626905 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-device-dir\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626927 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-sys\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626949 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89fde479-2004-4ff5-a4ed-1ad6180e0dda-hosts-file\") pod \"node-resolver-ddt89\" (UID: \"89fde479-2004-4ff5-a4ed-1ad6180e0dda\") " pod="openshift-dns/node-resolver-ddt89" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.626973 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc145ba6-3ca3-4293-850c-f3df88b34d9d-multus-daemon-config\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627000 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4qng\" (UniqueName: \"kubernetes.io/projected/b98c0ee8-844a-4f6c-89ce-f9492e87509b-kube-api-access-b4qng\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627025 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-etc-selinux\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627047 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-var-lib-kubelet\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627069 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds28m\" (UniqueName: \"kubernetes.io/projected/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-kube-api-access-ds28m\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627090 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc145ba6-3ca3-4293-850c-f3df88b34d9d-cni-binary-copy\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627112 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-node-log\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627171 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g4xh\" (UniqueName: \"kubernetes.io/projected/5480668d-f15c-4051-a9d4-a2ca9fefe4d3-kube-api-access-5g4xh\") pod \"node-ca-k62hz\" (UID: \"5480668d-f15c-4051-a9d4-a2ca9fefe4d3\") " pod="openshift-image-registry/node-ca-k62hz" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627201 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-var-lib-cni-bin\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627222 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-hostroot\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627240 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b98c0ee8-844a-4f6c-89ce-f9492e87509b-env-overrides\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.627276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627263 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5480668d-f15c-4051-a9d4-a2ca9fefe4d3-host\") pod \"node-ca-k62hz\" (UID: \"5480668d-f15c-4051-a9d4-a2ca9fefe4d3\") " pod="openshift-image-registry/node-ca-k62hz" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627288 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-systemd\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e58f844c-4b86-475d-b704-14d4e304d626-iptables-alerter-script\") pod \"iptables-alerter-dmmg6\" (UID: \"e58f844c-4b86-475d-b704-14d4e304d626\") " pod="openshift-network-operator/iptables-alerter-dmmg6" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627333 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-multus-socket-dir-parent\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627377 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-var-lib-cni-multus\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627402 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z6vj\" (UniqueName: \"kubernetes.io/projected/cc145ba6-3ca3-4293-850c-f3df88b34d9d-kube-api-access-6z6vj\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627425 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627465 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b98c0ee8-844a-4f6c-89ce-f9492e87509b-ovnkube-script-lib\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-sysconfig\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627545 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-host\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627697 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-systemd\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627731 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89fde479-2004-4ff5-a4ed-1ad6180e0dda-hosts-file\") pod \"node-resolver-ddt89\" (UID: \"89fde479-2004-4ff5-a4ed-1ad6180e0dda\") " pod="openshift-dns/node-resolver-ddt89" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627750 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-sysctl-conf\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627794 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-run-netns\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627823 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-var-lib-kubelet\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627847 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-run-ovn\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.628021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627874 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-log-socket\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627899 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-modprobe-d\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627925 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-lib-modules\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627950 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-tmp\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.627984 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt79v\" (UniqueName: \"kubernetes.io/projected/e58f844c-4b86-475d-b704-14d4e304d626-kube-api-access-tt79v\") pod \"iptables-alerter-dmmg6\" (UID: \"e58f844c-4b86-475d-b704-14d4e304d626\") " pod="openshift-network-operator/iptables-alerter-dmmg6" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628009 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-run-multus-certs\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628035 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/483e138d-49da-44ef-9e55-0d50f70ffb75-agent-certs\") pod \"konnectivity-agent-wjzd5\" (UID: \"483e138d-49da-44ef-9e55-0d50f70ffb75\") " pod="kube-system/konnectivity-agent-wjzd5" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628062 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89fde479-2004-4ff5-a4ed-1ad6180e0dda-tmp-dir\") pod \"node-resolver-ddt89\" (UID: \"89fde479-2004-4ff5-a4ed-1ad6180e0dda\") " pod="openshift-dns/node-resolver-ddt89" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628088 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-systemd-units\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628114 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628158 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nswrk\" (UniqueName: \"kubernetes.io/projected/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-kube-api-access-nswrk\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628189 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44bc3315-9704-4168-8d8e-869a507f3414-system-cni-dir\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628214 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-kubernetes\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628222 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-sys\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628242 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62e51f21-9e30-4c69-bca0-b1aca76a0a24-dbus\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628241 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2bn\" (UniqueName: \"kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn\") pod \"network-check-target-xsq5c\" (UID: \"3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f\") " pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.628316 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628427 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-var-lib-kubelet\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.628815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628430 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-sysctl-d\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628464 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-sysconfig\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628465 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-kubernetes\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628491 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e58f844c-4b86-475d-b704-14d4e304d626-host-slash\") pod \"iptables-alerter-dmmg6\" (UID: \"e58f844c-4b86-475d-b704-14d4e304d626\") " pod="openshift-network-operator/iptables-alerter-dmmg6" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628493 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44bc3315-9704-4168-8d8e-869a507f3414-system-cni-dir\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628658 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/44bc3315-9704-4168-8d8e-869a507f3414-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628714 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5480668d-f15c-4051-a9d4-a2ca9fefe4d3-host\") pod \"node-ca-k62hz\" (UID: \"5480668d-f15c-4051-a9d4-a2ca9fefe4d3\") " pod="openshift-image-registry/node-ca-k62hz" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628757 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-cnibin\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628788 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-multus-cni-dir\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628795 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628813 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-slash\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628829 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-modprobe-d\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628789 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89fde479-2004-4ff5-a4ed-1ad6180e0dda-tmp-dir\") pod \"node-resolver-ddt89\" (UID: \"89fde479-2004-4ff5-a4ed-1ad6180e0dda\") " pod="openshift-dns/node-resolver-ddt89" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.628876 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret podName:62e51f21-9e30-4c69-bca0-b1aca76a0a24 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:34.12882845 +0000 UTC m=+2.008217880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret") pod "global-pull-secret-syncer-lkhnc" (UID: "62e51f21-9e30-4c69-bca0-b1aca76a0a24") : object "kube-system"/"original-pull-secret" not registered Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628898 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44bc3315-9704-4168-8d8e-869a507f3414-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628926 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-tuned\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628936 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44bc3315-9704-4168-8d8e-869a507f3414-cni-binary-copy\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.629591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628953 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-run-systemd\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628980 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-etc-openvswitch\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628989 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-sysctl-conf\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628996 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e58f844c-4b86-475d-b704-14d4e304d626-iptables-alerter-script\") pod \"iptables-alerter-dmmg6\" (UID: \"e58f844c-4b86-475d-b704-14d4e304d626\") " pod="openshift-network-operator/iptables-alerter-dmmg6" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.628926 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-lib-modules\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629046 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-cni-netd\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629113 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44bc3315-9704-4168-8d8e-869a507f3414-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629192 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-socket-dir\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629219 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-registration-dir\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-run\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629287 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-sys-fs\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629330 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5480668d-f15c-4051-a9d4-a2ca9fefe4d3-serviceca\") pod \"node-ca-k62hz\" (UID: \"5480668d-f15c-4051-a9d4-a2ca9fefe4d3\") " pod="openshift-image-registry/node-ca-k62hz" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629333 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-run\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629364 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44bc3315-9704-4168-8d8e-869a507f3414-cnibin\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629404 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44bc3315-9704-4168-8d8e-869a507f3414-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629405 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44bc3315-9704-4168-8d8e-869a507f3414-cnibin\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629446 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62e51f21-9e30-4c69-bca0-b1aca76a0a24-kubelet-config\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:33.630452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629539 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6ccj\" (UniqueName: \"kubernetes.io/projected/89fde479-2004-4ff5-a4ed-1ad6180e0dda-kube-api-access-r6ccj\") pod \"node-resolver-ddt89\" (UID: \"89fde479-2004-4ff5-a4ed-1ad6180e0dda\") " pod="openshift-dns/node-resolver-ddt89" Sep 10 11:51:33.631286 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629572 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-run-k8s-cni-cncf-io\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.631286 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629587 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62e51f21-9e30-4c69-bca0-b1aca76a0a24-kubelet-config\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:33.631286 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-etc-kubernetes\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.631286 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629693 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5480668d-f15c-4051-a9d4-a2ca9fefe4d3-serviceca\") pod \"node-ca-k62hz\" (UID: \"5480668d-f15c-4051-a9d4-a2ca9fefe4d3\") " pod="openshift-image-registry/node-ca-k62hz" Sep 10 11:51:33.631286 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.629815 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/44bc3315-9704-4168-8d8e-869a507f3414-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.632303 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.632283 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-tmp\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.632437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.632420 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-etc-tuned\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.639541 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.639525 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 11:51:33.639623 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.639545 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 11:51:33.639623 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.639559 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nn2bn for pod openshift-network-diagnostics/network-check-target-xsq5c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:51:33.639623 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.639616 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn podName:3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f nodeName:}" failed. No retries permitted until 2026-09-10 11:51:34.139600594 +0000 UTC m=+2.018990047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nn2bn" (UniqueName: "kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn") pod "network-check-target-xsq5c" (UID: "3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:51:33.642408 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.642388 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds28m\" (UniqueName: \"kubernetes.io/projected/09e1491a-5e4a-4446-98cf-32c9fe5bd5b6-kube-api-access-ds28m\") pod \"tuned-4xtfr\" (UID: \"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6\") " pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.642484 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.642451 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt79v\" (UniqueName: \"kubernetes.io/projected/e58f844c-4b86-475d-b704-14d4e304d626-kube-api-access-tt79v\") pod \"iptables-alerter-dmmg6\" (UID: \"e58f844c-4b86-475d-b704-14d4e304d626\") " pod="openshift-network-operator/iptables-alerter-dmmg6" Sep 10 11:51:33.642669 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.642653 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jwqq\" (UniqueName: \"kubernetes.io/projected/44bc3315-9704-4168-8d8e-869a507f3414-kube-api-access-6jwqq\") pod \"multus-additional-cni-plugins-rxnr5\" (UID: \"44bc3315-9704-4168-8d8e-869a507f3414\") " pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.643111 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.643096 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6ccj\" (UniqueName: \"kubernetes.io/projected/89fde479-2004-4ff5-a4ed-1ad6180e0dda-kube-api-access-r6ccj\") pod \"node-resolver-ddt89\" (UID: \"89fde479-2004-4ff5-a4ed-1ad6180e0dda\") " pod="openshift-dns/node-resolver-ddt89" Sep 10 11:51:33.643514 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.643499 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g4xh\" (UniqueName: \"kubernetes.io/projected/5480668d-f15c-4051-a9d4-a2ca9fefe4d3-kube-api-access-5g4xh\") pod \"node-ca-k62hz\" (UID: \"5480668d-f15c-4051-a9d4-a2ca9fefe4d3\") " pod="openshift-image-registry/node-ca-k62hz" Sep 10 11:51:33.730725 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730692 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-var-lib-cni-multus\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.730725 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730723 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z6vj\" (UniqueName: \"kubernetes.io/projected/cc145ba6-3ca3-4293-850c-f3df88b34d9d-kube-api-access-6z6vj\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.730916 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730739 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.730916 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730755 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b98c0ee8-844a-4f6c-89ce-f9492e87509b-ovnkube-script-lib\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.730916 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730794 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.730916 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730807 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-var-lib-cni-multus\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.730916 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730817 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-run-netns\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.730916 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730839 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-var-lib-kubelet\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.730916 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730848 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-run-netns\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.730916 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730854 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-run-ovn\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.730916 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-log-socket\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.730916 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730897 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-run-ovn\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.730916 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-run-multus-certs\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.730916 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730913 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-var-lib-kubelet\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730920 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/483e138d-49da-44ef-9e55-0d50f70ffb75-agent-certs\") pod \"konnectivity-agent-wjzd5\" (UID: \"483e138d-49da-44ef-9e55-0d50f70ffb75\") " pod="kube-system/konnectivity-agent-wjzd5" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730919 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-log-socket\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730946 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-run-multus-certs\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-systemd-units\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.730991 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nswrk\" (UniqueName: \"kubernetes.io/projected/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-kube-api-access-nswrk\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731058 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-cnibin\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731061 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-systemd-units\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731072 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731084 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-multus-cni-dir\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731109 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-slash\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731134 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-run-systemd\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731168 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-multus-cni-dir\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731182 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-etc-openvswitch\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731156 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-cnibin\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-cni-netd\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731194 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-slash\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.731618 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731217 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-run-systemd\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731232 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-etc-openvswitch\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731264 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-socket-dir\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731274 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-cni-netd\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731317 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-registration-dir\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731357 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-sys-fs\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731365 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-socket-dir\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731386 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-run-k8s-cni-cncf-io\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731416 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-sys-fs\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731414 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-registration-dir\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-etc-kubernetes\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731467 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-run-k8s-cni-cncf-io\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731469 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b98c0ee8-844a-4f6c-89ce-f9492e87509b-ovnkube-script-lib\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-multus-conf-dir\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhpsj\" (UniqueName: \"kubernetes.io/projected/bb366139-ec7c-4ae4-be68-93027b09d18c-kube-api-access-bhpsj\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731541 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-etc-kubernetes\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.732368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731549 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/483e138d-49da-44ef-9e55-0d50f70ffb75-konnectivity-ca\") pod \"konnectivity-agent-wjzd5\" (UID: \"483e138d-49da-44ef-9e55-0d50f70ffb75\") " pod="kube-system/konnectivity-agent-wjzd5" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731575 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-multus-conf-dir\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731599 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-run-netns\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731617 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-var-lib-openvswitch\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731633 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-kubelet\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-cni-bin\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.731635 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731741 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-cni-bin\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731685 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-var-lib-openvswitch\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731709 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-kubelet\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731715 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-run-netns\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731718 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-system-cni-dir\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731679 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-system-cni-dir\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:33.731814 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs podName:bb366139-ec7c-4ae4-be68-93027b09d18c nodeName:}" failed. No retries permitted until 2026-09-10 11:51:34.231794916 +0000 UTC m=+2.111184348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs") pod "network-metrics-daemon-lnhpb" (UID: "bb366139-ec7c-4ae4-be68-93027b09d18c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-run-openvswitch\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731873 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-run-ovn-kubernetes\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731891 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-run-openvswitch\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731898 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b98c0ee8-844a-4f6c-89ce-f9492e87509b-ovnkube-config\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b98c0ee8-844a-4f6c-89ce-f9492e87509b-ovn-node-metrics-cert\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731923 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-host-run-ovn-kubernetes\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731951 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-device-dir\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.731979 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc145ba6-3ca3-4293-850c-f3df88b34d9d-multus-daemon-config\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/483e138d-49da-44ef-9e55-0d50f70ffb75-konnectivity-ca\") pod \"konnectivity-agent-wjzd5\" (UID: \"483e138d-49da-44ef-9e55-0d50f70ffb75\") " pod="kube-system/konnectivity-agent-wjzd5" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732004 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4qng\" (UniqueName: \"kubernetes.io/projected/b98c0ee8-844a-4f6c-89ce-f9492e87509b-kube-api-access-b4qng\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732066 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-device-dir\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732072 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-etc-selinux\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732126 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc145ba6-3ca3-4293-850c-f3df88b34d9d-cni-binary-copy\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732166 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-etc-selinux\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-node-log\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732209 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-var-lib-cni-bin\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-hostroot\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732243 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-host-var-lib-cni-bin\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732212 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b98c0ee8-844a-4f6c-89ce-f9492e87509b-node-log\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732260 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b98c0ee8-844a-4f6c-89ce-f9492e87509b-env-overrides\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732292 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-multus-socket-dir-parent\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.733648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732295 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-hostroot\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.734238 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732415 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b98c0ee8-844a-4f6c-89ce-f9492e87509b-ovnkube-config\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.734238 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732421 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc145ba6-3ca3-4293-850c-f3df88b34d9d-multus-socket-dir-parent\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.734238 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732609 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc145ba6-3ca3-4293-850c-f3df88b34d9d-multus-daemon-config\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.734238 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732653 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc145ba6-3ca3-4293-850c-f3df88b34d9d-cni-binary-copy\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.734238 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.732668 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b98c0ee8-844a-4f6c-89ce-f9492e87509b-env-overrides\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.734238 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.733577 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/483e138d-49da-44ef-9e55-0d50f70ffb75-agent-certs\") pod \"konnectivity-agent-wjzd5\" (UID: \"483e138d-49da-44ef-9e55-0d50f70ffb75\") " pod="kube-system/konnectivity-agent-wjzd5" Sep 10 11:51:33.734238 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.734052 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b98c0ee8-844a-4f6c-89ce-f9492e87509b-ovn-node-metrics-cert\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.758552 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.758525 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nswrk\" (UniqueName: \"kubernetes.io/projected/0b466c2f-9e9d-4d1b-8dcd-471cb402085c-kube-api-access-nswrk\") pod \"aws-ebs-csi-driver-node-5zgkv\" (UID: \"0b466c2f-9e9d-4d1b-8dcd-471cb402085c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.758880 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.758864 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhpsj\" (UniqueName: \"kubernetes.io/projected/bb366139-ec7c-4ae4-be68-93027b09d18c-kube-api-access-bhpsj\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:33.759216 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.759197 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z6vj\" (UniqueName: \"kubernetes.io/projected/cc145ba6-3ca3-4293-850c-f3df88b34d9d-kube-api-access-6z6vj\") pod \"multus-55lvl\" (UID: \"cc145ba6-3ca3-4293-850c-f3df88b34d9d\") " pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.760510 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.760492 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4qng\" (UniqueName: \"kubernetes.io/projected/b98c0ee8-844a-4f6c-89ce-f9492e87509b-kube-api-access-b4qng\") pod \"ovnkube-node-gnz8p\" (UID: \"b98c0ee8-844a-4f6c-89ce-f9492e87509b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.832259 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.832195 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" Sep 10 11:51:33.838901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.838884 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ddt89" Sep 10 11:51:33.866969 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.866944 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k62hz" Sep 10 11:51:33.872677 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.872524 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rxnr5" Sep 10 11:51:33.882964 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:33.882922 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5480668d_f15c_4051_a9d4_a2ca9fefe4d3.slice/crio-f00510e955a65724737b3d1940c3fb3a7a9b73686b7f8152e415e8c4e69230d0 WatchSource:0}: Error finding container f00510e955a65724737b3d1940c3fb3a7a9b73686b7f8152e415e8c4e69230d0: Status 404 returned error can't find the container with id f00510e955a65724737b3d1940c3fb3a7a9b73686b7f8152e415e8c4e69230d0 Sep 10 11:51:33.886771 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.886754 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 11:51:33.891520 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.891503 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dmmg6" Sep 10 11:51:33.900870 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:33.900847 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58f844c_4b86_475d_b704_14d4e304d626.slice/crio-52ec46b18cbb29e0fc453abc6b0e62a931c5f760a0acd666867ca2e562af8d88 WatchSource:0}: Error finding container 52ec46b18cbb29e0fc453abc6b0e62a931c5f760a0acd666867ca2e562af8d88: Status 404 returned error can't find the container with id 52ec46b18cbb29e0fc453abc6b0e62a931c5f760a0acd666867ca2e562af8d88 Sep 10 11:51:33.918252 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.918222 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-55lvl" Sep 10 11:51:33.925541 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:33.925521 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc145ba6_3ca3_4293_850c_f3df88b34d9d.slice/crio-d9527de41a2d12d0b6da058bbb295f19e172a3e526e4f355ac0f2dc0aa232697 WatchSource:0}: Error finding container d9527de41a2d12d0b6da058bbb295f19e172a3e526e4f355ac0f2dc0aa232697: Status 404 returned error can't find the container with id d9527de41a2d12d0b6da058bbb295f19e172a3e526e4f355ac0f2dc0aa232697 Sep 10 11:51:33.933496 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.933471 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:33.942726 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:33.942708 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98c0ee8_844a_4f6c_89ce_f9492e87509b.slice/crio-a976157ee5321b6b4b744964ca8d072b47b6b2bbb11444ff19c6ea8e0ec4c68c WatchSource:0}: Error finding container a976157ee5321b6b4b744964ca8d072b47b6b2bbb11444ff19c6ea8e0ec4c68c: Status 404 returned error can't find the container with id a976157ee5321b6b4b744964ca8d072b47b6b2bbb11444ff19c6ea8e0ec4c68c Sep 10 11:51:33.954353 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.954334 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wjzd5" Sep 10 11:51:33.958136 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:33.958115 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" Sep 10 11:51:33.962590 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:33.962568 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483e138d_49da_44ef_9e55_0d50f70ffb75.slice/crio-129d0b73710190dc92e2a21876ccb3b92219025a7303078875996d5c14547492 WatchSource:0}: Error finding container 129d0b73710190dc92e2a21876ccb3b92219025a7303078875996d5c14547492: Status 404 returned error can't find the container with id 129d0b73710190dc92e2a21876ccb3b92219025a7303078875996d5c14547492 Sep 10 11:51:33.967270 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:33.967249 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b466c2f_9e9d_4d1b_8dcd_471cb402085c.slice/crio-ecbfac3b1d4253b06629a9ba449d8fc5f7a86545989b78b45687f3a70423d9fd WatchSource:0}: Error finding container ecbfac3b1d4253b06629a9ba449d8fc5f7a86545989b78b45687f3a70423d9fd: Status 404 returned error can't find the container with id ecbfac3b1d4253b06629a9ba449d8fc5f7a86545989b78b45687f3a70423d9fd Sep 10 11:51:34.135051 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.134954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:34.135232 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:34.135085 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 11:51:34.135232 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:34.135137 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret podName:62e51f21-9e30-4c69-bca0-b1aca76a0a24 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:35.135123392 +0000 UTC m=+3.014512817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret") pod "global-pull-secret-syncer-lkhnc" (UID: "62e51f21-9e30-4c69-bca0-b1aca76a0a24") : object "kube-system"/"original-pull-secret" not registered Sep 10 11:51:34.235528 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.235495 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2bn\" (UniqueName: \"kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn\") pod \"network-check-target-xsq5c\" (UID: \"3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f\") " pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:34.235705 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.235547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:34.235705 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:34.235655 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:51:34.235705 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:34.235657 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 11:51:34.235705 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:34.235678 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 11:51:34.235705 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:34.235688 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nn2bn for pod openshift-network-diagnostics/network-check-target-xsq5c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:51:34.235925 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:34.235752 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs podName:bb366139-ec7c-4ae4-be68-93027b09d18c nodeName:}" failed. No retries permitted until 2026-09-10 11:51:35.235700706 +0000 UTC m=+3.115090134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs") pod "network-metrics-daemon-lnhpb" (UID: "bb366139-ec7c-4ae4-be68-93027b09d18c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:51:34.235925 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:34.235775 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn podName:3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f nodeName:}" failed. No retries permitted until 2026-09-10 11:51:35.23576254 +0000 UTC m=+3.115151957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nn2bn" (UniqueName: "kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn") pod "network-check-target-xsq5c" (UID: "3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:51:34.248709 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:34.248684 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e1491a_5e4a_4446_98cf_32c9fe5bd5b6.slice/crio-473f96303dd09902fbc46bafc8fc83eaedd4cab6c29c2bc3ffe490297150ca1a WatchSource:0}: Error finding container 473f96303dd09902fbc46bafc8fc83eaedd4cab6c29c2bc3ffe490297150ca1a: Status 404 returned error can't find the container with id 473f96303dd09902fbc46bafc8fc83eaedd4cab6c29c2bc3ffe490297150ca1a Sep 10 11:51:34.589646 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.589521 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 11:46:34 +0000 UTC" deadline="2028-04-18 06:01:25.367710357 +0000 UTC" Sep 10 11:51:34.589646 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.589595 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14058h9m50.778120599s" Sep 10 11:51:34.624050 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.624020 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:34.624254 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:34.624216 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:34.633286 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.633205 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" event={"ID":"0b466c2f-9e9d-4d1b-8dcd-471cb402085c","Type":"ContainerStarted","Data":"ecbfac3b1d4253b06629a9ba449d8fc5f7a86545989b78b45687f3a70423d9fd"} Sep 10 11:51:34.635029 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.634954 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wjzd5" event={"ID":"483e138d-49da-44ef-9e55-0d50f70ffb75","Type":"ContainerStarted","Data":"129d0b73710190dc92e2a21876ccb3b92219025a7303078875996d5c14547492"} Sep 10 11:51:34.637367 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.637253 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" event={"ID":"b98c0ee8-844a-4f6c-89ce-f9492e87509b","Type":"ContainerStarted","Data":"a976157ee5321b6b4b744964ca8d072b47b6b2bbb11444ff19c6ea8e0ec4c68c"} Sep 10 11:51:34.639580 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.639511 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k62hz" event={"ID":"5480668d-f15c-4051-a9d4-a2ca9fefe4d3","Type":"ContainerStarted","Data":"f00510e955a65724737b3d1940c3fb3a7a9b73686b7f8152e415e8c4e69230d0"} Sep 10 11:51:34.642943 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.642852 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-197.ec2.internal" event={"ID":"03d9f1651513641d55fc001cd4afe684","Type":"ContainerStarted","Data":"3e22f2afc4d121241c6f5ddd4fec7b6d5a161d0bdf261ef07e2fab81ae7daf15"} Sep 10 11:51:34.648414 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.648386 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal" event={"ID":"07dafb2c54e80bc28435cbc46f61ca4d","Type":"ContainerStarted","Data":"dce5c2f38b0719c50e84e7b0b9d5306f5777be98d731a3b99657f6819e1fac2b"} Sep 10 11:51:34.650865 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.650838 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" event={"ID":"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6","Type":"ContainerStarted","Data":"473f96303dd09902fbc46bafc8fc83eaedd4cab6c29c2bc3ffe490297150ca1a"} Sep 10 11:51:34.654485 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.654459 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-55lvl" event={"ID":"cc145ba6-3ca3-4293-850c-f3df88b34d9d","Type":"ContainerStarted","Data":"d9527de41a2d12d0b6da058bbb295f19e172a3e526e4f355ac0f2dc0aa232697"} Sep 10 11:51:34.656629 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.656605 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dmmg6" event={"ID":"e58f844c-4b86-475d-b704-14d4e304d626","Type":"ContainerStarted","Data":"52ec46b18cbb29e0fc453abc6b0e62a931c5f760a0acd666867ca2e562af8d88"} Sep 10 11:51:34.660107 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:34.660082 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rxnr5" event={"ID":"44bc3315-9704-4168-8d8e-869a507f3414","Type":"ContainerStarted","Data":"9715af30c8f92a5d3abb2af31ea8521494d26da1c288417489a91881163b99b4"} Sep 10 11:51:35.034700 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:35.034619 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 11:51:35.126933 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:51:35.126895 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fde479_2004_4ff5_a4ed_1ad6180e0dda.slice/crio-419d81574e7cb008f4663b8a7a7283fee85a1cc30017e870b04c63706c9574d1 WatchSource:0}: Error finding container 419d81574e7cb008f4663b8a7a7283fee85a1cc30017e870b04c63706c9574d1: Status 404 returned error can't find the container with id 419d81574e7cb008f4663b8a7a7283fee85a1cc30017e870b04c63706c9574d1 Sep 10 11:51:35.142738 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:35.142708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:35.142890 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:35.142872 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 11:51:35.142966 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:35.142948 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret podName:62e51f21-9e30-4c69-bca0-b1aca76a0a24 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:37.142928174 +0000 UTC m=+5.022317587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret") pod "global-pull-secret-syncer-lkhnc" (UID: "62e51f21-9e30-4c69-bca0-b1aca76a0a24") : object "kube-system"/"original-pull-secret" not registered Sep 10 11:51:35.243777 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:35.243143 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2bn\" (UniqueName: \"kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn\") pod \"network-check-target-xsq5c\" (UID: \"3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f\") " pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:35.243777 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:35.243226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:35.243777 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:35.243375 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:51:35.243777 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:35.243448 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs podName:bb366139-ec7c-4ae4-be68-93027b09d18c nodeName:}" failed. No retries permitted until 2026-09-10 11:51:37.243427764 +0000 UTC m=+5.122817195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs") pod "network-metrics-daemon-lnhpb" (UID: "bb366139-ec7c-4ae4-be68-93027b09d18c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:51:35.243777 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:35.243455 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 11:51:35.243777 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:35.243471 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 11:51:35.243777 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:35.243484 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nn2bn for pod openshift-network-diagnostics/network-check-target-xsq5c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:51:35.243777 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:35.243522 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn podName:3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f nodeName:}" failed. No retries permitted until 2026-09-10 11:51:37.243510241 +0000 UTC m=+5.122899671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nn2bn" (UniqueName: "kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn") pod "network-check-target-xsq5c" (UID: "3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:51:35.590561 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:35.590506 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 11:46:34 +0000 UTC" deadline="2028-06-06 02:33:49.331768056 +0000 UTC" Sep 10 11:51:35.590561 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:35.590560 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15230h42m13.741213037s" Sep 10 11:51:35.624375 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:35.624332 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:35.624568 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:35.624483 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:35.624650 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:35.624565 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:35.624650 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:35.624640 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:35.680642 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:35.680519 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ddt89" event={"ID":"89fde479-2004-4ff5-a4ed-1ad6180e0dda","Type":"ContainerStarted","Data":"419d81574e7cb008f4663b8a7a7283fee85a1cc30017e870b04c63706c9574d1"} Sep 10 11:51:36.090727 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:36.090364 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 11:51:36.625860 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:36.625789 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:36.626571 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:36.625922 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:37.161556 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:37.161516 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:37.161750 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:37.161652 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 11:51:37.161750 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:37.161715 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret podName:62e51f21-9e30-4c69-bca0-b1aca76a0a24 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:41.16169917 +0000 UTC m=+9.041088598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret") pod "global-pull-secret-syncer-lkhnc" (UID: "62e51f21-9e30-4c69-bca0-b1aca76a0a24") : object "kube-system"/"original-pull-secret" not registered Sep 10 11:51:37.263247 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:37.262421 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2bn\" (UniqueName: \"kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn\") pod \"network-check-target-xsq5c\" (UID: \"3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f\") " pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:37.263247 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:37.262484 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:37.263247 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:37.262628 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:51:37.263247 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:37.262691 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs podName:bb366139-ec7c-4ae4-be68-93027b09d18c nodeName:}" failed. No retries permitted until 2026-09-10 11:51:41.262671391 +0000 UTC m=+9.142060821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs") pod "network-metrics-daemon-lnhpb" (UID: "bb366139-ec7c-4ae4-be68-93027b09d18c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:51:37.263247 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:37.263104 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 11:51:37.263247 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:37.263127 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 11:51:37.263247 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:37.263141 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nn2bn for pod openshift-network-diagnostics/network-check-target-xsq5c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:51:37.263247 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:37.263197 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn podName:3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f nodeName:}" failed. No retries permitted until 2026-09-10 11:51:41.263171782 +0000 UTC m=+9.142561214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nn2bn" (UniqueName: "kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn") pod "network-check-target-xsq5c" (UID: "3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:51:37.623744 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:37.623693 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:37.623944 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:37.623862 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:37.624303 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:37.624281 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:37.624410 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:37.624380 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:38.624708 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:38.624415 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:38.624708 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:38.624558 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:39.624226 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:39.623619 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:39.624226 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:39.623763 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:39.624226 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:39.623826 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:39.624226 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:39.623896 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:40.624752 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:40.624305 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:40.624752 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:40.624428 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:41.196923 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:41.196887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:41.197102 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:41.197012 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 11:51:41.197102 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:41.197060 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret podName:62e51f21-9e30-4c69-bca0-b1aca76a0a24 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:49.197047049 +0000 UTC m=+17.076436472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret") pod "global-pull-secret-syncer-lkhnc" (UID: "62e51f21-9e30-4c69-bca0-b1aca76a0a24") : object "kube-system"/"original-pull-secret" not registered Sep 10 11:51:41.297688 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:41.297607 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2bn\" (UniqueName: \"kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn\") pod \"network-check-target-xsq5c\" (UID: \"3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f\") " pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:41.297688 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:41.297681 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:41.297860 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:41.297839 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:51:41.297912 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:41.297900 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs podName:bb366139-ec7c-4ae4-be68-93027b09d18c nodeName:}" failed. No retries permitted until 2026-09-10 11:51:49.297881508 +0000 UTC m=+17.177270923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs") pod "network-metrics-daemon-lnhpb" (UID: "bb366139-ec7c-4ae4-be68-93027b09d18c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:51:41.298353 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:41.298331 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 11:51:41.298403 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:41.298358 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 11:51:41.298403 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:41.298370 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nn2bn for pod openshift-network-diagnostics/network-check-target-xsq5c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:51:41.298467 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:41.298417 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn podName:3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f nodeName:}" failed. No retries permitted until 2026-09-10 11:51:49.298401999 +0000 UTC m=+17.177791414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nn2bn" (UniqueName: "kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn") pod "network-check-target-xsq5c" (UID: "3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:51:41.623486 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:41.623408 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:41.623486 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:41.623455 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:41.623690 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:41.623537 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:41.623690 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:41.623656 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:42.624797 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:42.624764 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:42.625132 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:42.624876 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:43.623795 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:43.623183 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:43.623795 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:43.623317 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:43.623795 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:43.623671 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:43.623795 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:43.623753 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:44.623371 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:44.623334 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:44.623822 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:44.623455 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:45.623703 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:45.623663 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:45.623703 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:45.623696 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:45.624174 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:45.623810 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:45.624174 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:45.623947 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:46.623977 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:46.623900 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:46.624425 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:46.624040 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:47.623372 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:47.623333 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:47.623512 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:47.623376 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:47.623512 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:47.623448 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:47.623597 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:47.623564 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:48.624222 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:48.623802 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:48.624222 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:48.623925 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:49.259256 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:49.259212 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:49.259448 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:49.259368 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 11:51:49.259523 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:49.259447 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret podName:62e51f21-9e30-4c69-bca0-b1aca76a0a24 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:05.259427182 +0000 UTC m=+33.138816607 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret") pod "global-pull-secret-syncer-lkhnc" (UID: "62e51f21-9e30-4c69-bca0-b1aca76a0a24") : object "kube-system"/"original-pull-secret" not registered Sep 10 11:51:49.360434 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:49.360393 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2bn\" (UniqueName: \"kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn\") pod \"network-check-target-xsq5c\" (UID: \"3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f\") " pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:49.360606 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:49.360451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:49.360606 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:49.360574 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 11:51:49.360606 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:49.360591 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 11:51:49.360606 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:49.360600 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nn2bn for pod openshift-network-diagnostics/network-check-target-xsq5c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:51:49.360819 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:49.360664 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn podName:3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f nodeName:}" failed. No retries permitted until 2026-09-10 11:52:05.360649706 +0000 UTC m=+33.240039121 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nn2bn" (UniqueName: "kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn") pod "network-check-target-xsq5c" (UID: "3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:51:49.361072 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:49.361050 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:51:49.361164 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:49.361114 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs podName:bb366139-ec7c-4ae4-be68-93027b09d18c nodeName:}" failed. No retries permitted until 2026-09-10 11:52:05.361098425 +0000 UTC m=+33.240487848 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs") pod "network-metrics-daemon-lnhpb" (UID: "bb366139-ec7c-4ae4-be68-93027b09d18c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:51:49.622996 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:49.622931 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:49.623216 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:49.622935 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:49.623216 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:49.623024 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:49.623216 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:49.623108 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:50.623187 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:50.623133 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:50.623731 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:50.623284 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:51.623855 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:51.623817 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:51.624336 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:51.623824 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:51.624336 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:51.623962 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:51.624336 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:51.624032 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:52.624208 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:52.624173 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:52.624595 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:52.624293 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:53.623969 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:53.623948 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:53.624058 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:53.624043 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:53.624111 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:53.624095 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:53.624216 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:53.624199 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:53.717981 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:53.717948 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-197.ec2.internal" event={"ID":"03d9f1651513641d55fc001cd4afe684","Type":"ContainerStarted","Data":"fd6ebc0aed7a49bf10f2098a3d20d6c31bb452302d5f42185f721eb78319eda0"} Sep 10 11:51:53.735190 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:53.734996 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-197.ec2.internal" podStartSLOduration=20.734979377 podStartE2EDuration="20.734979377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:51:53.734379357 +0000 UTC m=+21.613768791" watchObservedRunningTime="2026-09-10 11:51:53.734979377 +0000 UTC m=+21.614368809" Sep 10 11:51:54.624130 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.623965 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:54.624313 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:54.624247 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:54.721775 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.721732 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-55lvl" event={"ID":"cc145ba6-3ca3-4293-850c-f3df88b34d9d","Type":"ContainerStarted","Data":"8d8d1287fd2e9ce8a4c57fb2a4ec472a18b2678c04b8699a0d8d96a1254ba81c"} Sep 10 11:51:54.723303 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.723274 2582 generic.go:356] "Generic (PLEG): container finished" podID="44bc3315-9704-4168-8d8e-869a507f3414" containerID="47fd8dc4757492dc91ef4d09469cd6b0eb9c0377f536268777727a427b5885b5" exitCode=0 Sep 10 11:51:54.723423 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.723360 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rxnr5" event={"ID":"44bc3315-9704-4168-8d8e-869a507f3414","Type":"ContainerDied","Data":"47fd8dc4757492dc91ef4d09469cd6b0eb9c0377f536268777727a427b5885b5"} Sep 10 11:51:54.724869 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.724756 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ddt89" event={"ID":"89fde479-2004-4ff5-a4ed-1ad6180e0dda","Type":"ContainerStarted","Data":"48c1922e8811a615350afa5fc2793964da193076e2380aaec173ba4f7adfbbe5"} Sep 10 11:51:54.726315 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.726295 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" event={"ID":"0b466c2f-9e9d-4d1b-8dcd-471cb402085c","Type":"ContainerStarted","Data":"1aeb02840357705571eafa329413ef901cc203e517c0e040ab641a5d2a67c347"} Sep 10 11:51:54.727756 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.727728 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wjzd5" event={"ID":"483e138d-49da-44ef-9e55-0d50f70ffb75","Type":"ContainerStarted","Data":"5c219b641c2d38528852fa5068d1b9e5d11b5caa0eb1088f7f0b6ad0f58eddb8"} Sep 10 11:51:54.730440 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.730421 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 11:51:54.730781 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.730764 2582 generic.go:356] "Generic (PLEG): container finished" podID="b98c0ee8-844a-4f6c-89ce-f9492e87509b" containerID="674dea8893a8686d8be7d41591ef15ae3acbbfb11c0aa97ccd165edce28bee3b" exitCode=1 Sep 10 11:51:54.730848 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.730823 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" event={"ID":"b98c0ee8-844a-4f6c-89ce-f9492e87509b","Type":"ContainerStarted","Data":"13fc9ddcf2b5d638421417c1ac1668ef08e178b979e10ef7b16b606cb26edb1e"} Sep 10 11:51:54.730894 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.730870 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" event={"ID":"b98c0ee8-844a-4f6c-89ce-f9492e87509b","Type":"ContainerStarted","Data":"cbcaf678efff1642e950f2676fa091e92c9524cd44c169249942bbe3ef2009d0"} Sep 10 11:51:54.730936 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.730912 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" event={"ID":"b98c0ee8-844a-4f6c-89ce-f9492e87509b","Type":"ContainerStarted","Data":"e6f3d46e82620c5cecc2e059a9353e6464d3fd34679cf2676d507c5aa3718e97"} Sep 10 11:51:54.730968 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.730946 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" event={"ID":"b98c0ee8-844a-4f6c-89ce-f9492e87509b","Type":"ContainerStarted","Data":"973e5e90181945805afe284902e80bc799bba4c694a4ac3281e72d1d3949b635"} Sep 10 11:51:54.730997 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.730973 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" event={"ID":"b98c0ee8-844a-4f6c-89ce-f9492e87509b","Type":"ContainerDied","Data":"674dea8893a8686d8be7d41591ef15ae3acbbfb11c0aa97ccd165edce28bee3b"} Sep 10 11:51:54.731046 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.731007 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" event={"ID":"b98c0ee8-844a-4f6c-89ce-f9492e87509b","Type":"ContainerStarted","Data":"c8b64659d30b3c897bb51c04f39719b2e6552eaec0553c9d9f25c90bb9226c68"} Sep 10 11:51:54.732206 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.732183 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k62hz" event={"ID":"5480668d-f15c-4051-a9d4-a2ca9fefe4d3","Type":"ContainerStarted","Data":"3b113d3d3f81d6a3d9f5bb429b4074818047318e7c794ba60438d4fd9cede311"} Sep 10 11:51:54.733660 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.733639 2582 generic.go:356] "Generic (PLEG): container finished" podID="07dafb2c54e80bc28435cbc46f61ca4d" containerID="39296f16682361249f3966df10e8ba03666b8fa0c24fa873e69775436359be09" exitCode=0 Sep 10 11:51:54.733746 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.733699 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal" event={"ID":"07dafb2c54e80bc28435cbc46f61ca4d","Type":"ContainerDied","Data":"39296f16682361249f3966df10e8ba03666b8fa0c24fa873e69775436359be09"} Sep 10 11:51:54.735174 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.735137 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" event={"ID":"09e1491a-5e4a-4446-98cf-32c9fe5bd5b6","Type":"ContainerStarted","Data":"b7626a5971bdc82450c98121676ea7eafcc1e61c48fba4c44a8ea699190a3e97"} Sep 10 11:51:54.748386 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.748350 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-55lvl" podStartSLOduration=1.81322341 podStartE2EDuration="21.748338025s" totalImagesPullingTime="19.935114615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:51:54.748034961 +0000 UTC m=+22.627424393" watchObservedRunningTime="2026-09-10 11:51:54.748338025 +0000 UTC m=+22.627727486" Sep 10 11:51:54.767018 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.766981 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wjzd5" podStartSLOduration=2.273595466 podStartE2EDuration="21.766970522s" totalImagesPullingTime="19.493375056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:51:54.766580682 +0000 UTC m=+22.645970115" watchObservedRunningTime="2026-09-10 11:51:54.766970522 +0000 UTC m=+22.646359994" Sep 10 11:51:54.808426 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.808374 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4xtfr" podStartSLOduration=2.559120903 podStartE2EDuration="21.808354764s" totalImagesPullingTime="19.249233861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:51:54.807639768 +0000 UTC m=+22.687029202" watchObservedRunningTime="2026-09-10 11:51:54.808354764 +0000 UTC m=+22.687744197" Sep 10 11:51:54.856734 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.856685 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ddt89" podStartSLOduration=3.533986271 podStartE2EDuration="21.856666647s" totalImagesPullingTime="18.322680376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:51:54.834313628 +0000 UTC m=+22.713703060" watchObservedRunningTime="2026-09-10 11:51:54.856666647 +0000 UTC m=+22.736056079" Sep 10 11:51:54.856837 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:54.856759 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k62hz" podStartSLOduration=2.2859926870000002 podStartE2EDuration="21.856752404s" totalImagesPullingTime="19.570759717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:51:54.855517089 +0000 UTC m=+22.734906521" watchObservedRunningTime="2026-09-10 11:51:54.856752404 +0000 UTC m=+22.736141835" Sep 10 11:51:55.623495 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:55.623469 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:55.623610 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:55.623501 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:55.623668 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:55.623606 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:55.623751 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:55.623727 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:55.714973 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:55.714951 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 11:51:55.732595 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:55.732577 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wjzd5" Sep 10 11:51:55.739855 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:55.739805 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal" event={"ID":"07dafb2c54e80bc28435cbc46f61ca4d","Type":"ContainerStarted","Data":"32984d7b432ece33010ce4cab0ccbdeaf0da21901891f590198d53294acacd6a"} Sep 10 11:51:55.741620 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:55.741590 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dmmg6" event={"ID":"e58f844c-4b86-475d-b704-14d4e304d626","Type":"ContainerStarted","Data":"d79a035f37477378d4da8a1ff685dbb86de02c436a276ab94aa3cd9d886d1dc8"} Sep 10 11:51:55.743827 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:55.743802 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" event={"ID":"0b466c2f-9e9d-4d1b-8dcd-471cb402085c","Type":"ContainerStarted","Data":"fd5609306727f8c00e853ac489afb16210d863c0056103c00b1104d31cf32270"} Sep 10 11:51:55.759309 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:55.759268 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-197.ec2.internal" podStartSLOduration=22.759253947 podStartE2EDuration="22.759253947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:51:55.759204803 +0000 UTC m=+23.638594235" watchObservedRunningTime="2026-09-10 11:51:55.759253947 +0000 UTC m=+23.638643379" Sep 10 11:51:55.786159 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:55.786099 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dmmg6" podStartSLOduration=3.190147044 podStartE2EDuration="22.786082403s" totalImagesPullingTime="19.595935359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:51:55.78576283 +0000 UTC m=+23.665152264" watchObservedRunningTime="2026-09-10 11:51:55.786082403 +0000 UTC m=+23.665471837" Sep 10 11:51:56.589979 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:56.589416 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T11:51:55.714970255Z","UUID":"8c65f665-7b7a-4bef-8050-27a5ec2cf8ea","Handler":null,"Name":"","Endpoint":""} Sep 10 11:51:56.591382 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:56.591355 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 11:51:56.591382 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:56.591386 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 11:51:56.623653 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:56.623622 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:56.623775 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:56.623740 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:56.750111 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:56.750085 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 11:51:56.752746 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:56.752274 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" event={"ID":"b98c0ee8-844a-4f6c-89ce-f9492e87509b","Type":"ContainerStarted","Data":"5bd6ee97e21d4682ee620d1d2b47a446b8ed7f778c4ce2be81901c7706f85077"} Sep 10 11:51:57.623493 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:57.623458 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:57.623493 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:57.623480 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:57.623748 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:57.623583 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:57.623748 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:57.623694 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:57.756667 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:57.756630 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" event={"ID":"0b466c2f-9e9d-4d1b-8dcd-471cb402085c","Type":"ContainerStarted","Data":"2b76a6298ecac8c5d022e8a0014ff1bbc8b632bbccd388c499413da42342e23f"} Sep 10 11:51:57.777035 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:57.776991 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5zgkv" podStartSLOduration=2.931118439 podStartE2EDuration="24.776969625s" totalImagesPullingTime="21.845851186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:51:57.776605479 +0000 UTC m=+25.655994911" watchObservedRunningTime="2026-09-10 11:51:57.776969625 +0000 UTC m=+25.656359057" Sep 10 11:51:58.623333 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:58.623296 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:51:58.623509 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:58.623431 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:51:58.730937 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:58.730901 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wjzd5" Sep 10 11:51:58.731682 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:58.731663 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wjzd5" Sep 10 11:51:58.759750 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:58.759725 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wjzd5" Sep 10 11:51:59.623682 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:59.623465 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:51:59.623882 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:59.623488 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:51:59.623882 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:59.623730 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:51:59.623882 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:51:59.623772 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:51:59.765084 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:59.765059 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 11:51:59.765493 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:59.765471 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" event={"ID":"b98c0ee8-844a-4f6c-89ce-f9492e87509b","Type":"ContainerStarted","Data":"e139166656f7b52fca1a2551fdda9c38c7257a9505d779a9eca51dded12adec3"} Sep 10 11:51:59.765854 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:59.765722 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:59.765854 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:59.765803 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:51:59.766014 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:59.765892 2582 scope.go:117] "RemoveContainer" containerID="674dea8893a8686d8be7d41591ef15ae3acbbfb11c0aa97ccd165edce28bee3b" Sep 10 11:51:59.767652 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:59.767628 2582 generic.go:356] "Generic (PLEG): container finished" podID="44bc3315-9704-4168-8d8e-869a507f3414" containerID="73f9489ce95907faae40a75f6fe926f582ae3f774369dc9a0fcb81b0079dd35c" exitCode=0 Sep 10 11:51:59.768237 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:59.768192 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rxnr5" event={"ID":"44bc3315-9704-4168-8d8e-869a507f3414","Type":"ContainerDied","Data":"73f9489ce95907faae40a75f6fe926f582ae3f774369dc9a0fcb81b0079dd35c"} Sep 10 11:51:59.783081 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:51:59.783060 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:52:00.624053 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:00.624024 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:52:00.624323 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:00.624175 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:52:00.774300 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:00.774278 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 11:52:00.775444 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:00.775405 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" event={"ID":"b98c0ee8-844a-4f6c-89ce-f9492e87509b","Type":"ContainerStarted","Data":"e6828ab762771a4d93eb0ebbbeff11f298ffcd5c665477a21ba272d56279acc3"} Sep 10 11:52:00.775826 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:00.775750 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:52:00.794008 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:00.793984 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:52:00.810928 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:00.810886 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" podStartSLOduration=8.222963134 podStartE2EDuration="27.81087441s" totalImagesPullingTime="19.587911276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:00.809451755 +0000 UTC m=+28.688841187" watchObservedRunningTime="2026-09-10 11:52:00.81087441 +0000 UTC m=+28.690263843" Sep 10 11:52:00.900968 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:00.900894 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lnhpb"] Sep 10 11:52:00.901081 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:00.901019 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:52:00.901216 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:00.901103 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:52:00.901640 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:00.901614 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xsq5c"] Sep 10 11:52:00.901730 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:00.901719 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:52:00.901835 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:00.901815 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:52:00.903077 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:00.903054 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lkhnc"] Sep 10 11:52:00.903194 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:00.903181 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:52:00.903312 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:00.903290 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:52:01.779673 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:01.779634 2582 generic.go:356] "Generic (PLEG): container finished" podID="44bc3315-9704-4168-8d8e-869a507f3414" containerID="d864bd6e0ec55b0f53475dcf538c01f6e1e2ed6314ef1d2f78259b5047c66204" exitCode=0 Sep 10 11:52:01.780011 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:01.779712 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rxnr5" event={"ID":"44bc3315-9704-4168-8d8e-869a507f3414","Type":"ContainerDied","Data":"d864bd6e0ec55b0f53475dcf538c01f6e1e2ed6314ef1d2f78259b5047c66204"} Sep 10 11:52:02.624412 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:02.624190 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:52:02.624565 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:02.624218 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:52:02.624565 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:02.624495 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:52:02.624664 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:02.624249 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:52:02.624664 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:02.624600 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:52:02.624730 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:02.624706 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:52:04.623343 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:04.623311 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:52:04.623794 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:04.623319 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:52:04.623794 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:04.623409 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:52:04.623794 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:04.623426 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:52:04.623794 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:04.623514 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:52:04.623794 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:04.623578 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:52:04.788743 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:04.788714 2582 generic.go:356] "Generic (PLEG): container finished" podID="44bc3315-9704-4168-8d8e-869a507f3414" containerID="ffce82d80061b5f0a4e0afe64457b7441b906b2502dce5b3a38a54275500067d" exitCode=0 Sep 10 11:52:04.788893 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:04.788775 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rxnr5" event={"ID":"44bc3315-9704-4168-8d8e-869a507f3414","Type":"ContainerDied","Data":"ffce82d80061b5f0a4e0afe64457b7441b906b2502dce5b3a38a54275500067d"} Sep 10 11:52:05.280271 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:05.280241 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:52:05.280417 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:05.280401 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 11:52:05.280474 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:05.280457 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret podName:62e51f21-9e30-4c69-bca0-b1aca76a0a24 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:37.280442584 +0000 UTC m=+65.159831994 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret") pod "global-pull-secret-syncer-lkhnc" (UID: "62e51f21-9e30-4c69-bca0-b1aca76a0a24") : object "kube-system"/"original-pull-secret" not registered Sep 10 11:52:05.381041 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:05.381005 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2bn\" (UniqueName: \"kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn\") pod \"network-check-target-xsq5c\" (UID: \"3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f\") " pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:52:05.381041 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:05.381045 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:52:05.381302 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:05.381179 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:52:05.381302 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:05.381178 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 11:52:05.381302 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:05.381208 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 11:52:05.381302 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:05.381220 2582 projected.go:196] Error preparing data for projected volume kube-api-access-nn2bn for pod openshift-network-diagnostics/network-check-target-xsq5c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:52:05.381302 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:05.381236 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs podName:bb366139-ec7c-4ae4-be68-93027b09d18c nodeName:}" failed. No retries permitted until 2026-09-10 11:52:37.38122005 +0000 UTC m=+65.260609658 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs") pod "network-metrics-daemon-lnhpb" (UID: "bb366139-ec7c-4ae4-be68-93027b09d18c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 11:52:05.381302 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:05.381267 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn podName:3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f nodeName:}" failed. No retries permitted until 2026-09-10 11:52:37.381251393 +0000 UTC m=+65.260640812 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nn2bn" (UniqueName: "kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn") pod "network-check-target-xsq5c" (UID: "3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 11:52:06.623655 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:06.623095 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:52:06.623655 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:06.623095 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:52:06.623655 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:06.623237 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:52:06.623655 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:06.623243 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkhnc" podUID="62e51f21-9e30-4c69-bca0-b1aca76a0a24" Sep 10 11:52:06.623655 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:06.623358 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xsq5c" podUID="3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f" Sep 10 11:52:06.623655 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:06.623451 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lnhpb" podUID="bb366139-ec7c-4ae4-be68-93027b09d18c" Sep 10 11:52:07.400563 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.400531 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-197.ec2.internal" event="NodeReady" Sep 10 11:52:07.400704 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.400673 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 11:52:07.454638 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.454254 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8927q"] Sep 10 11:52:07.456206 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.456191 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8927q" Sep 10 11:52:07.459004 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.458986 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q76h9\"" Sep 10 11:52:07.459241 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.459218 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 11:52:07.459351 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.459317 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 11:52:07.460533 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.460512 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mk9ks"] Sep 10 11:52:07.462282 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.462244 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mk9ks" Sep 10 11:52:07.464665 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.464649 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 11:52:07.464883 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.464865 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 11:52:07.465375 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.465359 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2c8qf\"" Sep 10 11:52:07.465461 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.465420 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 11:52:07.471120 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.471102 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8927q"] Sep 10 11:52:07.478859 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.477737 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mk9ks"] Sep 10 11:52:07.597222 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.597160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1da2b0-9c77-4646-8dc1-5aad948d5380-config-volume\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:07.597355 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.597222 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f77tx\" (UniqueName: \"kubernetes.io/projected/fc1da2b0-9c77-4646-8dc1-5aad948d5380-kube-api-access-f77tx\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:07.597355 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.597270 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1da2b0-9c77-4646-8dc1-5aad948d5380-tmp-dir\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:07.597355 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.597301 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6tnt\" (UniqueName: \"kubernetes.io/projected/918767c5-ff0f-4d0a-9dbe-548d0a8463eb-kube-api-access-s6tnt\") pod \"ingress-canary-mk9ks\" (UID: \"918767c5-ff0f-4d0a-9dbe-548d0a8463eb\") " pod="openshift-ingress-canary/ingress-canary-mk9ks" Sep 10 11:52:07.597355 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.597326 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:07.597492 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.597381 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/918767c5-ff0f-4d0a-9dbe-548d0a8463eb-cert\") pod \"ingress-canary-mk9ks\" (UID: \"918767c5-ff0f-4d0a-9dbe-548d0a8463eb\") " pod="openshift-ingress-canary/ingress-canary-mk9ks" Sep 10 11:52:07.698299 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.698268 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1da2b0-9c77-4646-8dc1-5aad948d5380-config-volume\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:07.698722 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.698315 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f77tx\" (UniqueName: \"kubernetes.io/projected/fc1da2b0-9c77-4646-8dc1-5aad948d5380-kube-api-access-f77tx\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:07.698722 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.698352 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1da2b0-9c77-4646-8dc1-5aad948d5380-tmp-dir\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:07.698722 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.698374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6tnt\" (UniqueName: \"kubernetes.io/projected/918767c5-ff0f-4d0a-9dbe-548d0a8463eb-kube-api-access-s6tnt\") pod \"ingress-canary-mk9ks\" (UID: \"918767c5-ff0f-4d0a-9dbe-548d0a8463eb\") " pod="openshift-ingress-canary/ingress-canary-mk9ks" Sep 10 11:52:07.698722 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.698391 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:07.698722 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.698421 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/918767c5-ff0f-4d0a-9dbe-548d0a8463eb-cert\") pod \"ingress-canary-mk9ks\" (UID: \"918767c5-ff0f-4d0a-9dbe-548d0a8463eb\") " pod="openshift-ingress-canary/ingress-canary-mk9ks" Sep 10 11:52:07.698988 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:07.698952 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 11:52:07.698988 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.698954 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1da2b0-9c77-4646-8dc1-5aad948d5380-config-volume\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:07.699089 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:07.699012 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls podName:fc1da2b0-9c77-4646-8dc1-5aad948d5380 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:08.198993907 +0000 UTC m=+36.078383323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls") pod "dns-default-8927q" (UID: "fc1da2b0-9c77-4646-8dc1-5aad948d5380") : secret "dns-default-metrics-tls" not found Sep 10 11:52:07.699255 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.699228 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1da2b0-9c77-4646-8dc1-5aad948d5380-tmp-dir\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:07.703173 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.703132 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/918767c5-ff0f-4d0a-9dbe-548d0a8463eb-cert\") pod \"ingress-canary-mk9ks\" (UID: \"918767c5-ff0f-4d0a-9dbe-548d0a8463eb\") " pod="openshift-ingress-canary/ingress-canary-mk9ks" Sep 10 11:52:07.712835 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.712812 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6tnt\" (UniqueName: \"kubernetes.io/projected/918767c5-ff0f-4d0a-9dbe-548d0a8463eb-kube-api-access-s6tnt\") pod \"ingress-canary-mk9ks\" (UID: \"918767c5-ff0f-4d0a-9dbe-548d0a8463eb\") " pod="openshift-ingress-canary/ingress-canary-mk9ks" Sep 10 11:52:07.712932 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.712907 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f77tx\" (UniqueName: \"kubernetes.io/projected/fc1da2b0-9c77-4646-8dc1-5aad948d5380-kube-api-access-f77tx\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:07.772180 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:07.772110 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mk9ks" Sep 10 11:52:08.201797 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:08.201761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:08.201964 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:08.201907 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 11:52:08.202015 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:08.202007 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls podName:fc1da2b0-9c77-4646-8dc1-5aad948d5380 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:09.201992522 +0000 UTC m=+37.081381932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls") pod "dns-default-8927q" (UID: "fc1da2b0-9c77-4646-8dc1-5aad948d5380") : secret "dns-default-metrics-tls" not found Sep 10 11:52:08.623873 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:08.623645 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:52:08.624038 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:08.623645 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:52:08.624038 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:08.623644 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:52:08.627954 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:08.627769 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 11:52:08.627954 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:08.627769 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 11:52:08.627954 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:08.627817 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-69fjj\"" Sep 10 11:52:08.627954 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:08.627825 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 11:52:08.627954 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:08.627774 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 11:52:08.628281 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:08.628064 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zrvwr\"" Sep 10 11:52:09.211276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:09.211246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:09.211940 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:09.211396 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 11:52:09.211940 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:09.211462 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls podName:fc1da2b0-9c77-4646-8dc1-5aad948d5380 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:11.211441662 +0000 UTC m=+39.090831077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls") pod "dns-default-8927q" (UID: "fc1da2b0-9c77-4646-8dc1-5aad948d5380") : secret "dns-default-metrics-tls" not found Sep 10 11:52:09.344386 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:09.344318 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mk9ks"] Sep 10 11:52:09.376165 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:52:09.376127 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918767c5_ff0f_4d0a_9dbe_548d0a8463eb.slice/crio-bc0111a85d1acea25d174f22c19e6181cb36e631dbf5b54f9239e20cf2fe479f WatchSource:0}: Error finding container bc0111a85d1acea25d174f22c19e6181cb36e631dbf5b54f9239e20cf2fe479f: Status 404 returned error can't find the container with id bc0111a85d1acea25d174f22c19e6181cb36e631dbf5b54f9239e20cf2fe479f Sep 10 11:52:09.803536 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:09.803502 2582 generic.go:356] "Generic (PLEG): container finished" podID="44bc3315-9704-4168-8d8e-869a507f3414" containerID="75a2d32b8fb3fef1e849fc69df3b09d5690a9fe8ae4967898f88250228cc0953" exitCode=0 Sep 10 11:52:09.803692 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:09.803587 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rxnr5" event={"ID":"44bc3315-9704-4168-8d8e-869a507f3414","Type":"ContainerDied","Data":"75a2d32b8fb3fef1e849fc69df3b09d5690a9fe8ae4967898f88250228cc0953"} Sep 10 11:52:09.804729 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:09.804710 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mk9ks" event={"ID":"918767c5-ff0f-4d0a-9dbe-548d0a8463eb","Type":"ContainerStarted","Data":"bc0111a85d1acea25d174f22c19e6181cb36e631dbf5b54f9239e20cf2fe479f"} Sep 10 11:52:10.811824 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:10.811738 2582 generic.go:356] "Generic (PLEG): container finished" podID="44bc3315-9704-4168-8d8e-869a507f3414" containerID="4fb3d5fa536d50ebed495697960e1d68fecafe44ab58a68f3fef90ce8f342f33" exitCode=0 Sep 10 11:52:10.811824 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:10.811800 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rxnr5" event={"ID":"44bc3315-9704-4168-8d8e-869a507f3414","Type":"ContainerDied","Data":"4fb3d5fa536d50ebed495697960e1d68fecafe44ab58a68f3fef90ce8f342f33"} Sep 10 11:52:11.227910 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:11.227872 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:11.228077 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:11.228043 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 11:52:11.228123 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:11.228105 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls podName:fc1da2b0-9c77-4646-8dc1-5aad948d5380 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:15.228090928 +0000 UTC m=+43.107480337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls") pod "dns-default-8927q" (UID: "fc1da2b0-9c77-4646-8dc1-5aad948d5380") : secret "dns-default-metrics-tls" not found Sep 10 11:52:11.818711 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:11.818680 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rxnr5" event={"ID":"44bc3315-9704-4168-8d8e-869a507f3414","Type":"ContainerStarted","Data":"9f5a59ac20a638b9a347773668d7e1e6aa1e6900246d72bca68e0b83735163a1"} Sep 10 11:52:11.844671 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:11.844478 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rxnr5" podStartSLOduration=7.257057974 podStartE2EDuration="38.844457727s" totalImagesPullingTime="31.587399753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:11.844359316 +0000 UTC m=+39.723748749" watchObservedRunningTime="2026-09-10 11:52:11.844457727 +0000 UTC m=+39.723847161" Sep 10 11:52:13.825250 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:13.825213 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mk9ks" event={"ID":"918767c5-ff0f-4d0a-9dbe-548d0a8463eb","Type":"ContainerStarted","Data":"b93ec84a70f754d72c939cee1043464676a9e67dd6a4ed490be199320f957af5"} Sep 10 11:52:13.843958 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:13.843912 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mk9ks" podStartSLOduration=3.383057363 podStartE2EDuration="6.843899595s" totalImagesPullingTime="3.460842232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:13.842522497 +0000 UTC m=+41.721911929" watchObservedRunningTime="2026-09-10 11:52:13.843899595 +0000 UTC m=+41.723289026" Sep 10 11:52:15.259362 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:15.259323 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:15.259732 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:15.259449 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 11:52:15.259732 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:15.259499 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls podName:fc1da2b0-9c77-4646-8dc1-5aad948d5380 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:23.259485956 +0000 UTC m=+51.138875367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls") pod "dns-default-8927q" (UID: "fc1da2b0-9c77-4646-8dc1-5aad948d5380") : secret "dns-default-metrics-tls" not found Sep 10 11:52:23.312500 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:23.312457 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:23.312915 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:23.312623 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 11:52:23.312915 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:23.312689 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls podName:fc1da2b0-9c77-4646-8dc1-5aad948d5380 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:39.312674394 +0000 UTC m=+67.192063803 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls") pod "dns-default-8927q" (UID: "fc1da2b0-9c77-4646-8dc1-5aad948d5380") : secret "dns-default-metrics-tls" not found Sep 10 11:52:32.792895 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:32.792868 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnz8p" Sep 10 11:52:37.313952 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.313900 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:52:37.317748 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.317729 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 11:52:37.326905 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.326878 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e51f21-9e30-4c69-bca0-b1aca76a0a24-original-pull-secret\") pod \"global-pull-secret-syncer-lkhnc\" (UID: \"62e51f21-9e30-4c69-bca0-b1aca76a0a24\") " pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:52:37.414772 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.414746 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2bn\" (UniqueName: \"kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn\") pod \"network-check-target-xsq5c\" (UID: \"3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f\") " pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:52:37.414874 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.414781 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:52:37.417817 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.417798 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 11:52:37.417878 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.417844 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 11:52:37.425246 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:37.425222 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 11:52:37.425318 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:37.425289 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs podName:bb366139-ec7c-4ae4-be68-93027b09d18c nodeName:}" failed. No retries permitted until 2026-09-10 11:53:41.425275028 +0000 UTC m=+129.304664450 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs") pod "network-metrics-daemon-lnhpb" (UID: "bb366139-ec7c-4ae4-be68-93027b09d18c") : secret "metrics-daemon-secret" not found Sep 10 11:52:37.427487 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.427475 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 11:52:37.435621 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.435605 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkhnc" Sep 10 11:52:37.438186 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.437865 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn2bn\" (UniqueName: \"kubernetes.io/projected/3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f-kube-api-access-nn2bn\") pod \"network-check-target-xsq5c\" (UID: \"3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f\") " pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:52:37.454204 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.454186 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-69fjj\"" Sep 10 11:52:37.461639 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.461607 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:52:37.704456 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.704425 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lkhnc"] Sep 10 11:52:37.708164 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:52:37.708120 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e51f21_9e30_4c69_bca0_b1aca76a0a24.slice/crio-cc5b4b79123757598b89b0030a8b67accb5730ee808c11cf2b4bcc14f305f971 WatchSource:0}: Error finding container cc5b4b79123757598b89b0030a8b67accb5730ee808c11cf2b4bcc14f305f971: Status 404 returned error can't find the container with id cc5b4b79123757598b89b0030a8b67accb5730ee808c11cf2b4bcc14f305f971 Sep 10 11:52:37.740397 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.740375 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xsq5c"] Sep 10 11:52:37.743539 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:52:37.743519 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3899cdfd_1bf1_4c95_a0a7_5bddf3235a6f.slice/crio-5d22aa18659779281e319f2ce501f017b1dd007e344f64028de693012cdcf7ef WatchSource:0}: Error finding container 5d22aa18659779281e319f2ce501f017b1dd007e344f64028de693012cdcf7ef: Status 404 returned error can't find the container with id 5d22aa18659779281e319f2ce501f017b1dd007e344f64028de693012cdcf7ef Sep 10 11:52:37.880997 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.880960 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xsq5c" event={"ID":"3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f","Type":"ContainerStarted","Data":"5d22aa18659779281e319f2ce501f017b1dd007e344f64028de693012cdcf7ef"} Sep 10 11:52:37.882191 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:37.882166 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lkhnc" event={"ID":"62e51f21-9e30-4c69-bca0-b1aca76a0a24","Type":"ContainerStarted","Data":"cc5b4b79123757598b89b0030a8b67accb5730ee808c11cf2b4bcc14f305f971"} Sep 10 11:52:39.328693 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:39.328646 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:52:39.329094 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:39.328830 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 11:52:39.329094 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:39.328901 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls podName:fc1da2b0-9c77-4646-8dc1-5aad948d5380 nodeName:}" failed. No retries permitted until 2026-09-10 11:53:11.328885391 +0000 UTC m=+99.208274801 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls") pod "dns-default-8927q" (UID: "fc1da2b0-9c77-4646-8dc1-5aad948d5380") : secret "dns-default-metrics-tls" not found Sep 10 11:52:42.896819 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:42.896776 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xsq5c" event={"ID":"3899cdfd-1bf1-4c95-a0a7-5bddf3235a6f","Type":"ContainerStarted","Data":"750f59def775dbc319ab3b8c5755b5163f0ae11ac303a29a3f5f349379b2bd34"} Sep 10 11:52:42.897325 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:42.896975 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:52:42.898286 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:42.898266 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lkhnc" event={"ID":"62e51f21-9e30-4c69-bca0-b1aca76a0a24","Type":"ContainerStarted","Data":"2cd47a6bb741686da0e8b0f015c33b62b7cf4be6f4ffa253832af6d911ef5e46"} Sep 10 11:52:42.931511 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:42.931465 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lkhnc" podStartSLOduration=65.272690814 podStartE2EDuration="1m9.931451802s" totalImagesPullingTime="4.658760988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:42.930624788 +0000 UTC m=+70.810014221" watchObservedRunningTime="2026-09-10 11:52:42.931451802 +0000 UTC m=+70.810841234" Sep 10 11:52:42.931800 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:42.931780 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xsq5c" podStartSLOduration=65.313171453 podStartE2EDuration="1m9.931772772s" totalImagesPullingTime="4.618601319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:42.913752526 +0000 UTC m=+70.793141957" watchObservedRunningTime="2026-09-10 11:52:42.931772772 +0000 UTC m=+70.811162203" Sep 10 11:52:45.736863 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.736824 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lwrbw"] Sep 10 11:52:45.749501 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.749474 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lwrbw" Sep 10 11:52:45.752086 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.752062 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lwrbw"] Sep 10 11:52:45.753137 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.753112 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 10 11:52:45.754224 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.754205 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 10 11:52:45.754224 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.754226 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-j4xqz\"" Sep 10 11:52:45.772211 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.772181 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96vxg\" (UniqueName: \"kubernetes.io/projected/ccf57465-583d-4ca7-98f4-fb61688670cf-kube-api-access-96vxg\") pod \"volume-data-source-validator-695c895bd6-lwrbw\" (UID: \"ccf57465-583d-4ca7-98f4-fb61688670cf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lwrbw" Sep 10 11:52:45.858409 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.858377 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d8fcd69-6lg7t"] Sep 10 11:52:45.859978 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.859964 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:45.864058 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.864037 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 11:52:45.864174 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.864038 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sbl2s\"" Sep 10 11:52:45.864174 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.864071 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 11:52:45.864595 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.864576 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 11:52:45.871810 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.871792 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 11:52:45.872552 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.872532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96vxg\" (UniqueName: \"kubernetes.io/projected/ccf57465-583d-4ca7-98f4-fb61688670cf-kube-api-access-96vxg\") pod \"volume-data-source-validator-695c895bd6-lwrbw\" (UID: \"ccf57465-583d-4ca7-98f4-fb61688670cf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lwrbw" Sep 10 11:52:45.879048 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.879030 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d8fcd69-6lg7t"] Sep 10 11:52:45.893920 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.893901 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96vxg\" (UniqueName: \"kubernetes.io/projected/ccf57465-583d-4ca7-98f4-fb61688670cf-kube-api-access-96vxg\") pod \"volume-data-source-validator-695c895bd6-lwrbw\" (UID: \"ccf57465-583d-4ca7-98f4-fb61688670cf\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lwrbw" Sep 10 11:52:45.919105 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.919077 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-kwhz7"] Sep 10 11:52:45.920648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.920634 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:45.923341 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.923318 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 11:52:45.923434 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.923369 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 10 11:52:45.923434 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.923319 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 10 11:52:45.923434 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.923412 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j5287\"" Sep 10 11:52:45.923964 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.923951 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 11:52:45.929463 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.929426 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 10 11:52:45.933935 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.933918 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-kwhz7"] Sep 10 11:52:45.973216 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4258c9d2-a577-427c-b0b1-5b6f373eaef2-ca-trust-extracted\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:45.973216 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973214 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-certificates\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:45.973395 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973235 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4258c9d2-a577-427c-b0b1-5b6f373eaef2-installation-pull-secrets\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:45.973395 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973250 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/313f87bb-5d68-4539-a59b-226c262d5309-tmp\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:45.973395 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973327 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4258c9d2-a577-427c-b0b1-5b6f373eaef2-image-registry-private-configuration\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:45.973395 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973365 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313f87bb-5d68-4539-a59b-226c262d5309-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:45.973395 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973384 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gcqs\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-kube-api-access-2gcqs\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:45.973550 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973425 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:45.973550 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973441 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4258c9d2-a577-427c-b0b1-5b6f373eaef2-trusted-ca\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:45.973550 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973455 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-bound-sa-token\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:45.973550 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973470 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/313f87bb-5d68-4539-a59b-226c262d5309-snapshots\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:45.973550 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973504 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313f87bb-5d68-4539-a59b-226c262d5309-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:45.973550 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973533 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc5gr\" (UniqueName: \"kubernetes.io/projected/313f87bb-5d68-4539-a59b-226c262d5309-kube-api-access-nc5gr\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:45.973725 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:45.973596 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/313f87bb-5d68-4539-a59b-226c262d5309-serving-cert\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.059509 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.059433 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lwrbw" Sep 10 11:52:46.074316 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074290 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313f87bb-5d68-4539-a59b-226c262d5309-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.074439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074324 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gcqs\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-kube-api-access-2gcqs\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.074439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.074514 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:46.074442 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 11:52:46.074514 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:46.074457 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d8fcd69-6lg7t: secret "image-registry-tls" not found Sep 10 11:52:46.074574 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:46.074514 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls podName:4258c9d2-a577-427c-b0b1-5b6f373eaef2 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:46.574493377 +0000 UTC m=+74.453882810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls") pod "image-registry-6d8fcd69-6lg7t" (UID: "4258c9d2-a577-427c-b0b1-5b6f373eaef2") : secret "image-registry-tls" not found Sep 10 11:52:46.074574 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074535 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4258c9d2-a577-427c-b0b1-5b6f373eaef2-trusted-ca\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.074574 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074565 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-bound-sa-token\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.074777 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/313f87bb-5d68-4539-a59b-226c262d5309-snapshots\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.074777 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074613 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313f87bb-5d68-4539-a59b-226c262d5309-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.074777 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074640 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc5gr\" (UniqueName: \"kubernetes.io/projected/313f87bb-5d68-4539-a59b-226c262d5309-kube-api-access-nc5gr\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.074777 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074699 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/313f87bb-5d68-4539-a59b-226c262d5309-serving-cert\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.074777 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074736 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4258c9d2-a577-427c-b0b1-5b6f373eaef2-ca-trust-extracted\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.074777 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-certificates\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.075024 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4258c9d2-a577-427c-b0b1-5b6f373eaef2-installation-pull-secrets\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.075024 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074825 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/313f87bb-5d68-4539-a59b-226c262d5309-tmp\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.075024 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.074865 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4258c9d2-a577-427c-b0b1-5b6f373eaef2-image-registry-private-configuration\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.075186 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.075041 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313f87bb-5d68-4539-a59b-226c262d5309-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.075380 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.075360 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/313f87bb-5d68-4539-a59b-226c262d5309-snapshots\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.075446 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.075368 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/313f87bb-5d68-4539-a59b-226c262d5309-tmp\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.075784 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.075595 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4258c9d2-a577-427c-b0b1-5b6f373eaef2-ca-trust-extracted\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.075784 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.075607 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313f87bb-5d68-4539-a59b-226c262d5309-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.075784 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.075711 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4258c9d2-a577-427c-b0b1-5b6f373eaef2-trusted-ca\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.076009 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.075922 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-certificates\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.077732 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.077710 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/313f87bb-5d68-4539-a59b-226c262d5309-serving-cert\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.077821 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.077753 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4258c9d2-a577-427c-b0b1-5b6f373eaef2-image-registry-private-configuration\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.078133 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.078118 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4258c9d2-a577-427c-b0b1-5b6f373eaef2-installation-pull-secrets\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.084297 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.084271 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gcqs\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-kube-api-access-2gcqs\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.084558 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.084541 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc5gr\" (UniqueName: \"kubernetes.io/projected/313f87bb-5d68-4539-a59b-226c262d5309-kube-api-access-nc5gr\") pod \"insights-operator-76bcd6ccf5-kwhz7\" (UID: \"313f87bb-5d68-4539-a59b-226c262d5309\") " pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.086400 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.086380 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-bound-sa-token\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.230191 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.230138 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" Sep 10 11:52:46.337352 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.337295 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lwrbw"] Sep 10 11:52:46.340222 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:52:46.340201 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf57465_583d_4ca7_98f4_fb61688670cf.slice/crio-0c1282ba6ab0d148a88bf597d05af177af7c7de544e884a3b86a411f2cd9fe21 WatchSource:0}: Error finding container 0c1282ba6ab0d148a88bf597d05af177af7c7de544e884a3b86a411f2cd9fe21: Status 404 returned error can't find the container with id 0c1282ba6ab0d148a88bf597d05af177af7c7de544e884a3b86a411f2cd9fe21 Sep 10 11:52:46.501969 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.501939 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-kwhz7"] Sep 10 11:52:46.505016 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:52:46.504989 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313f87bb_5d68_4539_a59b_226c262d5309.slice/crio-c668b5858a603e9250777b5d1a2e142305b188b0bbac137f0e3efc4513e56943 WatchSource:0}: Error finding container c668b5858a603e9250777b5d1a2e142305b188b0bbac137f0e3efc4513e56943: Status 404 returned error can't find the container with id c668b5858a603e9250777b5d1a2e142305b188b0bbac137f0e3efc4513e56943 Sep 10 11:52:46.579292 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.579261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:46.579424 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:46.579383 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 11:52:46.579424 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:46.579395 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d8fcd69-6lg7t: secret "image-registry-tls" not found Sep 10 11:52:46.579505 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:46.579440 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls podName:4258c9d2-a577-427c-b0b1-5b6f373eaef2 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:47.579427029 +0000 UTC m=+75.458816440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls") pod "image-registry-6d8fcd69-6lg7t" (UID: "4258c9d2-a577-427c-b0b1-5b6f373eaef2") : secret "image-registry-tls" not found Sep 10 11:52:46.914519 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.914476 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" event={"ID":"313f87bb-5d68-4539-a59b-226c262d5309","Type":"ContainerStarted","Data":"c668b5858a603e9250777b5d1a2e142305b188b0bbac137f0e3efc4513e56943"} Sep 10 11:52:46.915854 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:46.915831 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lwrbw" event={"ID":"ccf57465-583d-4ca7-98f4-fb61688670cf","Type":"ContainerStarted","Data":"0c1282ba6ab0d148a88bf597d05af177af7c7de544e884a3b86a411f2cd9fe21"} Sep 10 11:52:47.587819 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:47.587784 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:47.587947 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:47.587932 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 11:52:47.588006 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:47.587950 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d8fcd69-6lg7t: secret "image-registry-tls" not found Sep 10 11:52:47.588006 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:47.588003 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls podName:4258c9d2-a577-427c-b0b1-5b6f373eaef2 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:49.587989258 +0000 UTC m=+77.467378668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls") pod "image-registry-6d8fcd69-6lg7t" (UID: "4258c9d2-a577-427c-b0b1-5b6f373eaef2") : secret "image-registry-tls" not found Sep 10 11:52:47.920938 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:47.920897 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lwrbw" event={"ID":"ccf57465-583d-4ca7-98f4-fb61688670cf","Type":"ContainerStarted","Data":"3b15abcd005d81aae73523c3ef99c0542a9ba4b1cfbdba1a5eaf11dcd287c13d"} Sep 10 11:52:47.940715 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:47.940671 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lwrbw" podStartSLOduration=1.8871332920000001 podStartE2EDuration="2.940651027s" totalImagesPullingTime="1.053517735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:47.939857446 +0000 UTC m=+75.819246881" watchObservedRunningTime="2026-09-10 11:52:47.940651027 +0000 UTC m=+75.820040459" Sep 10 11:52:49.604084 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:49.604048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:49.604550 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:49.604229 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 11:52:49.604550 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:49.604252 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d8fcd69-6lg7t: secret "image-registry-tls" not found Sep 10 11:52:49.604550 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:49.604318 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls podName:4258c9d2-a577-427c-b0b1-5b6f373eaef2 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:53.604298802 +0000 UTC m=+81.483688229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls") pod "image-registry-6d8fcd69-6lg7t" (UID: "4258c9d2-a577-427c-b0b1-5b6f373eaef2") : secret "image-registry-tls" not found Sep 10 11:52:49.927662 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:49.927583 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" event={"ID":"313f87bb-5d68-4539-a59b-226c262d5309","Type":"ContainerStarted","Data":"87143ecf10a4c8351e66c98f40624a07dad604c454d271816176454230e9e69f"} Sep 10 11:52:49.946892 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:49.946851 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" podStartSLOduration=2.536845283 podStartE2EDuration="4.94683626s" totalImagesPullingTime="2.409990977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:49.945823994 +0000 UTC m=+77.825213426" watchObservedRunningTime="2026-09-10 11:52:49.94683626 +0000 UTC m=+77.826225675" Sep 10 11:52:50.415727 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.415692 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wrvdx"] Sep 10 11:52:50.418877 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.418850 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:50.421861 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.421834 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s4qzc\"" Sep 10 11:52:50.421971 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.421839 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 11:52:50.422979 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.422962 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 11:52:50.440221 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.440194 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wrvdx"] Sep 10 11:52:50.511348 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.511324 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a6812340-a7eb-46fc-95de-912e76a052d7-crio-socket\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:50.511470 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.511353 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6812340-a7eb-46fc-95de-912e76a052d7-data-volume\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:50.511470 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.511386 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:50.511470 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.511420 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5x4r\" (UniqueName: \"kubernetes.io/projected/a6812340-a7eb-46fc-95de-912e76a052d7-kube-api-access-g5x4r\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:50.511575 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.511470 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:50.612681 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.612642 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:50.612681 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.612679 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5x4r\" (UniqueName: \"kubernetes.io/projected/a6812340-a7eb-46fc-95de-912e76a052d7-kube-api-access-g5x4r\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:50.613100 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.612709 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:50.613100 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:50.612765 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 11:52:50.613100 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.612792 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a6812340-a7eb-46fc-95de-912e76a052d7-crio-socket\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:50.613100 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.612814 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6812340-a7eb-46fc-95de-912e76a052d7-data-volume\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:50.613100 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:50.612838 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm podName:a6812340-a7eb-46fc-95de-912e76a052d7 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:51.112816419 +0000 UTC m=+78.992205855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrvdx" (UID: "a6812340-a7eb-46fc-95de-912e76a052d7") : configmap "kube-rbac-proxy" not found Sep 10 11:52:50.613100 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.612885 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a6812340-a7eb-46fc-95de-912e76a052d7-crio-socket\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:50.613100 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:50.612899 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 11:52:50.613100 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:50.612952 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls podName:a6812340-a7eb-46fc-95de-912e76a052d7 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:51.112939383 +0000 UTC m=+78.992328793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrvdx" (UID: "a6812340-a7eb-46fc-95de-912e76a052d7") : secret "insights-runtime-extractor-tls" not found Sep 10 11:52:50.613391 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.613122 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a6812340-a7eb-46fc-95de-912e76a052d7-data-volume\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:50.629250 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:50.629229 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5x4r\" (UniqueName: \"kubernetes.io/projected/a6812340-a7eb-46fc-95de-912e76a052d7-kube-api-access-g5x4r\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:51.117456 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:51.117416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:51.117456 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:51.117460 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:51.117678 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:51.117529 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 11:52:51.117678 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:51.117551 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 11:52:51.117678 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:51.117601 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm podName:a6812340-a7eb-46fc-95de-912e76a052d7 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:52.117580277 +0000 UTC m=+79.996969690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrvdx" (UID: "a6812340-a7eb-46fc-95de-912e76a052d7") : configmap "kube-rbac-proxy" not found Sep 10 11:52:51.117678 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:51.117621 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls podName:a6812340-a7eb-46fc-95de-912e76a052d7 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:52.117611966 +0000 UTC m=+79.997001376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrvdx" (UID: "a6812340-a7eb-46fc-95de-912e76a052d7") : secret "insights-runtime-extractor-tls" not found Sep 10 11:52:52.124720 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:52.124677 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:52.124720 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:52.124729 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:52.125133 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:52.124805 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 11:52:52.125133 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:52.124817 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 11:52:52.125133 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:52.124865 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls podName:a6812340-a7eb-46fc-95de-912e76a052d7 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:54.124852746 +0000 UTC m=+82.004242157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrvdx" (UID: "a6812340-a7eb-46fc-95de-912e76a052d7") : secret "insights-runtime-extractor-tls" not found Sep 10 11:52:52.125133 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:52.124877 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm podName:a6812340-a7eb-46fc-95de-912e76a052d7 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:54.124871267 +0000 UTC m=+82.004260677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrvdx" (UID: "a6812340-a7eb-46fc-95de-912e76a052d7") : configmap "kube-rbac-proxy" not found Sep 10 11:52:52.142519 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:52.142499 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ddt89_89fde479-2004-4ff5-a4ed-1ad6180e0dda/dns-node-resolver/0.log" Sep 10 11:52:53.544044 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:53.543940 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k62hz_5480668d-f15c-4051-a9d4-a2ca9fefe4d3/node-ca/0.log" Sep 10 11:52:53.635395 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:53.635358 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:52:53.635528 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:53.635510 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 11:52:53.635528 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:53.635522 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d8fcd69-6lg7t: secret "image-registry-tls" not found Sep 10 11:52:53.635615 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:53.635573 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls podName:4258c9d2-a577-427c-b0b1-5b6f373eaef2 nodeName:}" failed. No retries permitted until 2026-09-10 11:53:01.635559739 +0000 UTC m=+89.514949149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls") pod "image-registry-6d8fcd69-6lg7t" (UID: "4258c9d2-a577-427c-b0b1-5b6f373eaef2") : secret "image-registry-tls" not found Sep 10 11:52:54.139291 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:54.139260 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:54.139445 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:54.139342 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:54.139445 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:54.139412 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 11:52:54.139445 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:54.139416 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 11:52:54.139555 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:54.139462 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm podName:a6812340-a7eb-46fc-95de-912e76a052d7 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:58.139448982 +0000 UTC m=+86.018838391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrvdx" (UID: "a6812340-a7eb-46fc-95de-912e76a052d7") : configmap "kube-rbac-proxy" not found Sep 10 11:52:54.139555 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:54.139475 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls podName:a6812340-a7eb-46fc-95de-912e76a052d7 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:58.139469561 +0000 UTC m=+86.018858971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrvdx" (UID: "a6812340-a7eb-46fc-95de-912e76a052d7") : secret "insights-runtime-extractor-tls" not found Sep 10 11:52:54.341009 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:54.340985 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mk9ks_918767c5-ff0f-4d0a-9dbe-548d0a8463eb/serve-healthcheck-canary/0.log" Sep 10 11:52:58.171162 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:58.171109 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:58.171517 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:52:58.171183 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:52:58.171517 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:58.171237 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 11:52:58.171517 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:58.171315 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm podName:a6812340-a7eb-46fc-95de-912e76a052d7 nodeName:}" failed. No retries permitted until 2026-09-10 11:53:06.171297261 +0000 UTC m=+94.050686688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wrvdx" (UID: "a6812340-a7eb-46fc-95de-912e76a052d7") : configmap "kube-rbac-proxy" not found Sep 10 11:52:58.171517 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:58.171344 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 11:52:58.171517 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:52:58.171407 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls podName:a6812340-a7eb-46fc-95de-912e76a052d7 nodeName:}" failed. No retries permitted until 2026-09-10 11:53:06.171393746 +0000 UTC m=+94.050783156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wrvdx" (UID: "a6812340-a7eb-46fc-95de-912e76a052d7") : secret "insights-runtime-extractor-tls" not found Sep 10 11:53:01.697445 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:01.697400 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:53:01.699795 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:01.699778 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls\") pod \"image-registry-6d8fcd69-6lg7t\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:53:01.769323 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:01.769298 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:53:02.043087 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:02.043014 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d8fcd69-6lg7t"] Sep 10 11:53:02.046014 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:02.045987 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4258c9d2_a577_427c_b0b1_5b6f373eaef2.slice/crio-e57ab2ab5aeec0f2b51bc8b251f7c5777437ad77a10846d6bb120afb5abdcd56 WatchSource:0}: Error finding container e57ab2ab5aeec0f2b51bc8b251f7c5777437ad77a10846d6bb120afb5abdcd56: Status 404 returned error can't find the container with id e57ab2ab5aeec0f2b51bc8b251f7c5777437ad77a10846d6bb120afb5abdcd56 Sep 10 11:53:02.970547 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:02.970514 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" event={"ID":"4258c9d2-a577-427c-b0b1-5b6f373eaef2","Type":"ContainerStarted","Data":"dd9436da02859697cde4da3c6a2421ec104fce5d15f258589330650fb2916313"} Sep 10 11:53:02.970936 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:02.970559 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" event={"ID":"4258c9d2-a577-427c-b0b1-5b6f373eaef2","Type":"ContainerStarted","Data":"e57ab2ab5aeec0f2b51bc8b251f7c5777437ad77a10846d6bb120afb5abdcd56"} Sep 10 11:53:02.970936 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:02.970645 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:53:02.994605 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:02.994538 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" podStartSLOduration=17.994524539 podStartE2EDuration="17.994524539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:02.993813994 +0000 UTC m=+90.873203427" watchObservedRunningTime="2026-09-10 11:53:02.994524539 +0000 UTC m=+90.873913971" Sep 10 11:53:06.232414 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:06.232375 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:53:06.232414 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:06.232420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:53:06.232866 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:06.232846 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a6812340-a7eb-46fc-95de-912e76a052d7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:53:06.234778 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:06.234762 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a6812340-a7eb-46fc-95de-912e76a052d7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wrvdx\" (UID: \"a6812340-a7eb-46fc-95de-912e76a052d7\") " pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:53:06.328598 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:06.328554 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wrvdx" Sep 10 11:53:06.603726 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:06.603637 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wrvdx"] Sep 10 11:53:06.606969 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:06.606943 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6812340_a7eb_46fc_95de_912e76a052d7.slice/crio-9d5472b7d7ddb63a24a31c9f53896de36ef7c2133290e663c85984d853fc2faf WatchSource:0}: Error finding container 9d5472b7d7ddb63a24a31c9f53896de36ef7c2133290e663c85984d853fc2faf: Status 404 returned error can't find the container with id 9d5472b7d7ddb63a24a31c9f53896de36ef7c2133290e663c85984d853fc2faf Sep 10 11:53:06.983081 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:06.983046 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wrvdx" event={"ID":"a6812340-a7eb-46fc-95de-912e76a052d7","Type":"ContainerStarted","Data":"102ad5b30bf19087467a27c23368fd27a81f83701536689353db2dec06372639"} Sep 10 11:53:06.983262 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:06.983110 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wrvdx" event={"ID":"a6812340-a7eb-46fc-95de-912e76a052d7","Type":"ContainerStarted","Data":"9d5472b7d7ddb63a24a31c9f53896de36ef7c2133290e663c85984d853fc2faf"} Sep 10 11:53:07.987425 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:07.987392 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wrvdx" event={"ID":"a6812340-a7eb-46fc-95de-912e76a052d7","Type":"ContainerStarted","Data":"79e551a953682ff47cc4503d2886ddf41ae04f766fe100b3cc36039ecb08ed65"} Sep 10 11:53:09.995235 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:09.995201 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wrvdx" event={"ID":"a6812340-a7eb-46fc-95de-912e76a052d7","Type":"ContainerStarted","Data":"633fab61644d3a3c4191f5c930852acdd72eb9f92ca59a8520a1aafbed59b5a6"} Sep 10 11:53:10.013412 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:10.013369 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wrvdx" podStartSLOduration=17.77617863 podStartE2EDuration="20.013354993s" totalImagesPullingTime="2.237176363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:52:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:10.013088668 +0000 UTC m=+97.892478100" watchObservedRunningTime="2026-09-10 11:53:10.013354993 +0000 UTC m=+97.892744424" Sep 10 11:53:11.370504 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:11.370459 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:53:11.372815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:11.372796 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1da2b0-9c77-4646-8dc1-5aad948d5380-metrics-tls\") pod \"dns-default-8927q\" (UID: \"fc1da2b0-9c77-4646-8dc1-5aad948d5380\") " pod="openshift-dns/dns-default-8927q" Sep 10 11:53:11.669214 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:11.669128 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q76h9\"" Sep 10 11:53:11.676903 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:11.676882 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8927q" Sep 10 11:53:11.949708 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:11.949670 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8927q"] Sep 10 11:53:11.952593 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:11.952563 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1da2b0_9c77_4646_8dc1_5aad948d5380.slice/crio-8f16e1015c13b31479acdb7fcc90bded1ef60626237ba62af442bae67be4cc27 WatchSource:0}: Error finding container 8f16e1015c13b31479acdb7fcc90bded1ef60626237ba62af442bae67be4cc27: Status 404 returned error can't find the container with id 8f16e1015c13b31479acdb7fcc90bded1ef60626237ba62af442bae67be4cc27 Sep 10 11:53:12.005434 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:12.005403 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8927q" event={"ID":"fc1da2b0-9c77-4646-8dc1-5aad948d5380","Type":"ContainerStarted","Data":"8f16e1015c13b31479acdb7fcc90bded1ef60626237ba62af442bae67be4cc27"} Sep 10 11:53:13.903089 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:13.903062 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xsq5c" Sep 10 11:53:15.016093 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:15.016059 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8927q" event={"ID":"fc1da2b0-9c77-4646-8dc1-5aad948d5380","Type":"ContainerStarted","Data":"a0f8c20eebb5d4f948e986c1caae00f8541b689906cd3e1154de3c06420089d4"} Sep 10 11:53:15.016529 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:15.016114 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8927q" event={"ID":"fc1da2b0-9c77-4646-8dc1-5aad948d5380","Type":"ContainerStarted","Data":"1e0e14275b569f2f53ff9418e7b40b0d6310bb4d23f84e57840e352a1632ad1b"} Sep 10 11:53:15.016529 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:15.016184 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8927q" Sep 10 11:53:15.034966 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:15.034919 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8927q" podStartSLOduration=65.958256183 podStartE2EDuration="1m8.034902177s" totalImagesPullingTime="2.076645994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:15.034726933 +0000 UTC m=+102.914116365" watchObservedRunningTime="2026-09-10 11:53:15.034902177 +0000 UTC m=+102.914291602" Sep 10 11:53:17.964854 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:17.964818 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-xhwjx"] Sep 10 11:53:17.967974 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:17.967955 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-xhwjx" Sep 10 11:53:17.970421 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:17.970398 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 10 11:53:17.970531 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:17.970484 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 10 11:53:17.971584 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:17.971569 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5g6cs\"" Sep 10 11:53:17.982019 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:17.979285 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k"] Sep 10 11:53:17.982019 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:17.990351 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k" Sep 10 11:53:17.994591 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:17.994567 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 11:53:17.994870 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:17.994853 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-xhwjx"] Sep 10 11:53:17.996409 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:17.996389 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k"] Sep 10 11:53:18.000961 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.000913 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 11:53:18.000961 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.000940 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 11:53:18.001999 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.001192 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 10 11:53:18.001999 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.001395 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2s4zl\"" Sep 10 11:53:18.061279 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.061254 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d8fcd69-6lg7t"] Sep 10 11:53:18.064652 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.064624 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gmvjk"] Sep 10 11:53:18.065891 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.065866 2582 patch_prober.go:28] interesting pod/image-registry-6d8fcd69-6lg7t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 11:53:18.066000 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.065906 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" podUID="4258c9d2-a577-427c-b0b1-5b6f373eaef2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 11:53:18.067737 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.067723 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gmvjk" Sep 10 11:53:18.072182 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.072163 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 10 11:53:18.072458 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.072439 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 10 11:53:18.072997 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.072977 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4srk2\"" Sep 10 11:53:18.082512 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.082493 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gmvjk"] Sep 10 11:53:18.123133 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.123104 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be59f2c-4e40-447c-884e-f32ee3c9af46-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-xhwjx\" (UID: \"1be59f2c-4e40-447c-884e-f32ee3c9af46\") " pod="openshift-network-console/networking-console-plugin-79977f9844-xhwjx" Sep 10 11:53:18.123283 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.123138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7954ff8a-86a5-484f-85c1-b03833adbc34-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dccbb8d46-sr95k\" (UID: \"7954ff8a-86a5-484f-85c1-b03833adbc34\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k" Sep 10 11:53:18.123283 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.123189 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxfjb\" (UniqueName: \"kubernetes.io/projected/7954ff8a-86a5-484f-85c1-b03833adbc34-kube-api-access-gxfjb\") pod \"managed-serviceaccount-addon-agent-dccbb8d46-sr95k\" (UID: \"7954ff8a-86a5-484f-85c1-b03833adbc34\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k" Sep 10 11:53:18.123283 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.123218 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1be59f2c-4e40-447c-884e-f32ee3c9af46-nginx-conf\") pod \"networking-console-plugin-79977f9844-xhwjx\" (UID: \"1be59f2c-4e40-447c-884e-f32ee3c9af46\") " pod="openshift-network-console/networking-console-plugin-79977f9844-xhwjx" Sep 10 11:53:18.166606 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.166581 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-9lg8f"] Sep 10 11:53:18.169683 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.169669 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d684479cb-4mz2c"] Sep 10 11:53:18.169817 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.169800 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-9lg8f" Sep 10 11:53:18.172727 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.172710 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.173811 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.173787 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 11:53:18.173901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.173786 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9mhlv\"" Sep 10 11:53:18.175216 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.175201 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 11:53:18.191358 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.191331 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d684479cb-4mz2c"] Sep 10 11:53:18.192462 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.192443 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-9lg8f"] Sep 10 11:53:18.224028 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.223972 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz84t\" (UniqueName: \"kubernetes.io/projected/1b5f2151-3c8b-4297-9d61-3f978d112702-kube-api-access-xz84t\") pod \"migrator-fdbb8fdff-gmvjk\" (UID: \"1b5f2151-3c8b-4297-9d61-3f978d112702\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gmvjk" Sep 10 11:53:18.224028 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.224000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be59f2c-4e40-447c-884e-f32ee3c9af46-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-xhwjx\" (UID: \"1be59f2c-4e40-447c-884e-f32ee3c9af46\") " pod="openshift-network-console/networking-console-plugin-79977f9844-xhwjx" Sep 10 11:53:18.224028 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.224021 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7954ff8a-86a5-484f-85c1-b03833adbc34-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dccbb8d46-sr95k\" (UID: \"7954ff8a-86a5-484f-85c1-b03833adbc34\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k" Sep 10 11:53:18.224274 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.224081 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxfjb\" (UniqueName: \"kubernetes.io/projected/7954ff8a-86a5-484f-85c1-b03833adbc34-kube-api-access-gxfjb\") pod \"managed-serviceaccount-addon-agent-dccbb8d46-sr95k\" (UID: \"7954ff8a-86a5-484f-85c1-b03833adbc34\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k" Sep 10 11:53:18.224274 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.224182 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1be59f2c-4e40-447c-884e-f32ee3c9af46-nginx-conf\") pod \"networking-console-plugin-79977f9844-xhwjx\" (UID: \"1be59f2c-4e40-447c-884e-f32ee3c9af46\") " pod="openshift-network-console/networking-console-plugin-79977f9844-xhwjx" Sep 10 11:53:18.224900 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.224879 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1be59f2c-4e40-447c-884e-f32ee3c9af46-nginx-conf\") pod \"networking-console-plugin-79977f9844-xhwjx\" (UID: \"1be59f2c-4e40-447c-884e-f32ee3c9af46\") " pod="openshift-network-console/networking-console-plugin-79977f9844-xhwjx" Sep 10 11:53:18.226703 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.226686 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1be59f2c-4e40-447c-884e-f32ee3c9af46-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-xhwjx\" (UID: \"1be59f2c-4e40-447c-884e-f32ee3c9af46\") " pod="openshift-network-console/networking-console-plugin-79977f9844-xhwjx" Sep 10 11:53:18.226812 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.226796 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7954ff8a-86a5-484f-85c1-b03833adbc34-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dccbb8d46-sr95k\" (UID: \"7954ff8a-86a5-484f-85c1-b03833adbc34\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k" Sep 10 11:53:18.238441 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.238424 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxfjb\" (UniqueName: \"kubernetes.io/projected/7954ff8a-86a5-484f-85c1-b03833adbc34-kube-api-access-gxfjb\") pod \"managed-serviceaccount-addon-agent-dccbb8d46-sr95k\" (UID: \"7954ff8a-86a5-484f-85c1-b03833adbc34\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k" Sep 10 11:53:18.301536 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.301496 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-xhwjx" Sep 10 11:53:18.320315 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.320296 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k" Sep 10 11:53:18.325399 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.325379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa968658-2fe1-4464-a872-a8eab3c88061-registry-tls\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.325504 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.325417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa968658-2fe1-4464-a872-a8eab3c88061-bound-sa-token\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.325504 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.325441 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa968658-2fe1-4464-a872-a8eab3c88061-image-registry-private-configuration\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.325601 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.325561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa968658-2fe1-4464-a872-a8eab3c88061-installation-pull-secrets\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.325669 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.325646 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr8v8\" (UniqueName: \"kubernetes.io/projected/a88118da-c9a3-4593-8a83-5137bdd8984a-kube-api-access-zr8v8\") pod \"downloads-85d98c998f-9lg8f\" (UID: \"a88118da-c9a3-4593-8a83-5137bdd8984a\") " pod="openshift-console/downloads-85d98c998f-9lg8f" Sep 10 11:53:18.325707 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.325677 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa968658-2fe1-4464-a872-a8eab3c88061-trusted-ca\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.325707 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.325702 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz84t\" (UniqueName: \"kubernetes.io/projected/1b5f2151-3c8b-4297-9d61-3f978d112702-kube-api-access-xz84t\") pod \"migrator-fdbb8fdff-gmvjk\" (UID: \"1b5f2151-3c8b-4297-9d61-3f978d112702\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gmvjk" Sep 10 11:53:18.325795 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.325721 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa968658-2fe1-4464-a872-a8eab3c88061-registry-certificates\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.325795 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.325736 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpl6s\" (UniqueName: \"kubernetes.io/projected/aa968658-2fe1-4464-a872-a8eab3c88061-kube-api-access-jpl6s\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.325951 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.325937 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa968658-2fe1-4464-a872-a8eab3c88061-ca-trust-extracted\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.336842 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.336815 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz84t\" (UniqueName: \"kubernetes.io/projected/1b5f2151-3c8b-4297-9d61-3f978d112702-kube-api-access-xz84t\") pod \"migrator-fdbb8fdff-gmvjk\" (UID: \"1b5f2151-3c8b-4297-9d61-3f978d112702\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gmvjk" Sep 10 11:53:18.376469 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.376438 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gmvjk" Sep 10 11:53:18.427819 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.427308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa968658-2fe1-4464-a872-a8eab3c88061-installation-pull-secrets\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.427819 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.427374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr8v8\" (UniqueName: \"kubernetes.io/projected/a88118da-c9a3-4593-8a83-5137bdd8984a-kube-api-access-zr8v8\") pod \"downloads-85d98c998f-9lg8f\" (UID: \"a88118da-c9a3-4593-8a83-5137bdd8984a\") " pod="openshift-console/downloads-85d98c998f-9lg8f" Sep 10 11:53:18.427819 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.427400 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa968658-2fe1-4464-a872-a8eab3c88061-trusted-ca\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.427819 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.427431 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa968658-2fe1-4464-a872-a8eab3c88061-registry-certificates\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.427819 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.427454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpl6s\" (UniqueName: \"kubernetes.io/projected/aa968658-2fe1-4464-a872-a8eab3c88061-kube-api-access-jpl6s\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.427819 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.427514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa968658-2fe1-4464-a872-a8eab3c88061-ca-trust-extracted\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.427819 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.427549 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa968658-2fe1-4464-a872-a8eab3c88061-registry-tls\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.427819 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.427578 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa968658-2fe1-4464-a872-a8eab3c88061-bound-sa-token\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.427819 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.427602 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa968658-2fe1-4464-a872-a8eab3c88061-image-registry-private-configuration\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.428789 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.428738 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa968658-2fe1-4464-a872-a8eab3c88061-ca-trust-extracted\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.432832 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.430366 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa968658-2fe1-4464-a872-a8eab3c88061-registry-certificates\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.432832 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.430479 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa968658-2fe1-4464-a872-a8eab3c88061-trusted-ca\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.432832 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.430597 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa968658-2fe1-4464-a872-a8eab3c88061-installation-pull-secrets\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.432832 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.431054 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa968658-2fe1-4464-a872-a8eab3c88061-registry-tls\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.437560 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.435734 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa968658-2fe1-4464-a872-a8eab3c88061-image-registry-private-configuration\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.438282 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.438255 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr8v8\" (UniqueName: \"kubernetes.io/projected/a88118da-c9a3-4593-8a83-5137bdd8984a-kube-api-access-zr8v8\") pod \"downloads-85d98c998f-9lg8f\" (UID: \"a88118da-c9a3-4593-8a83-5137bdd8984a\") " pod="openshift-console/downloads-85d98c998f-9lg8f" Sep 10 11:53:18.439766 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.439743 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpl6s\" (UniqueName: \"kubernetes.io/projected/aa968658-2fe1-4464-a872-a8eab3c88061-kube-api-access-jpl6s\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.440652 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.440627 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa968658-2fe1-4464-a872-a8eab3c88061-bound-sa-token\") pod \"image-registry-d684479cb-4mz2c\" (UID: \"aa968658-2fe1-4464-a872-a8eab3c88061\") " pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.478792 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.478730 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-9lg8f" Sep 10 11:53:18.483805 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.483791 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:18.582981 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.582958 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-xhwjx"] Sep 10 11:53:18.585507 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:18.585481 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be59f2c_4e40_447c_884e_f32ee3c9af46.slice/crio-e27b6c309feb3643595a7237fe54cb26ea27e7881f6232bb7c8324a215c752b1 WatchSource:0}: Error finding container e27b6c309feb3643595a7237fe54cb26ea27e7881f6232bb7c8324a215c752b1: Status 404 returned error can't find the container with id e27b6c309feb3643595a7237fe54cb26ea27e7881f6232bb7c8324a215c752b1 Sep 10 11:53:18.602707 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.602688 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k"] Sep 10 11:53:18.605294 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:18.605265 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7954ff8a_86a5_484f_85c1_b03833adbc34.slice/crio-d64a6f8d781bc9f85be1f683606d333642b7b34b38aef07098baac9223e35fd4 WatchSource:0}: Error finding container d64a6f8d781bc9f85be1f683606d333642b7b34b38aef07098baac9223e35fd4: Status 404 returned error can't find the container with id d64a6f8d781bc9f85be1f683606d333642b7b34b38aef07098baac9223e35fd4 Sep 10 11:53:18.647779 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.647757 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gmvjk"] Sep 10 11:53:18.650102 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:18.650080 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b5f2151_3c8b_4297_9d61_3f978d112702.slice/crio-3f87070fd3156324614130cb314cc9b51c3cf43d37700ee43de88302fb3a8fd2 WatchSource:0}: Error finding container 3f87070fd3156324614130cb314cc9b51c3cf43d37700ee43de88302fb3a8fd2: Status 404 returned error can't find the container with id 3f87070fd3156324614130cb314cc9b51c3cf43d37700ee43de88302fb3a8fd2 Sep 10 11:53:18.768570 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.768515 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-9lg8f"] Sep 10 11:53:18.771086 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:18.771065 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88118da_c9a3_4593_8a83_5137bdd8984a.slice/crio-b9d655447a21f988fb6a156a3886770fb30275317021addf5b79e85db623162c WatchSource:0}: Error finding container b9d655447a21f988fb6a156a3886770fb30275317021addf5b79e85db623162c: Status 404 returned error can't find the container with id b9d655447a21f988fb6a156a3886770fb30275317021addf5b79e85db623162c Sep 10 11:53:18.775920 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:18.775902 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d684479cb-4mz2c"] Sep 10 11:53:18.778276 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:18.778256 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa968658_2fe1_4464_a872_a8eab3c88061.slice/crio-af6f704f8dbf0257bad702d0ba8c3472a7e43bff5b2ef00e6d635803c4e156ff WatchSource:0}: Error finding container af6f704f8dbf0257bad702d0ba8c3472a7e43bff5b2ef00e6d635803c4e156ff: Status 404 returned error can't find the container with id af6f704f8dbf0257bad702d0ba8c3472a7e43bff5b2ef00e6d635803c4e156ff Sep 10 11:53:19.029815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:19.029736 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d684479cb-4mz2c" event={"ID":"aa968658-2fe1-4464-a872-a8eab3c88061","Type":"ContainerStarted","Data":"96644a036a13084aef7411bcf8006457a09068768ba578301206b410652bc99b"} Sep 10 11:53:19.029815 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:19.029791 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d684479cb-4mz2c" event={"ID":"aa968658-2fe1-4464-a872-a8eab3c88061","Type":"ContainerStarted","Data":"af6f704f8dbf0257bad702d0ba8c3472a7e43bff5b2ef00e6d635803c4e156ff"} Sep 10 11:53:19.030313 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:19.029866 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:19.031086 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:19.031060 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k" event={"ID":"7954ff8a-86a5-484f-85c1-b03833adbc34","Type":"ContainerStarted","Data":"d64a6f8d781bc9f85be1f683606d333642b7b34b38aef07098baac9223e35fd4"} Sep 10 11:53:19.032297 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:19.032275 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-xhwjx" event={"ID":"1be59f2c-4e40-447c-884e-f32ee3c9af46","Type":"ContainerStarted","Data":"e27b6c309feb3643595a7237fe54cb26ea27e7881f6232bb7c8324a215c752b1"} Sep 10 11:53:19.033439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:19.033417 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gmvjk" event={"ID":"1b5f2151-3c8b-4297-9d61-3f978d112702","Type":"ContainerStarted","Data":"3f87070fd3156324614130cb314cc9b51c3cf43d37700ee43de88302fb3a8fd2"} Sep 10 11:53:19.034584 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:19.034563 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-9lg8f" event={"ID":"a88118da-c9a3-4593-8a83-5137bdd8984a","Type":"ContainerStarted","Data":"b9d655447a21f988fb6a156a3886770fb30275317021addf5b79e85db623162c"} Sep 10 11:53:19.051748 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:19.051707 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d684479cb-4mz2c" podStartSLOduration=1.05169318 podStartE2EDuration="1.05169318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:19.050900819 +0000 UTC m=+106.930290252" watchObservedRunningTime="2026-09-10 11:53:19.05169318 +0000 UTC m=+106.931082659" Sep 10 11:53:20.231932 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:20.230929 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz"] Sep 10 11:53:20.237177 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:20.237097 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz" Sep 10 11:53:20.240690 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:20.240503 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 10 11:53:20.240690 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:20.240640 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mwnjl\"" Sep 10 11:53:20.250986 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:20.250923 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz"] Sep 10 11:53:20.343240 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:20.343195 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a847cea1-5939-4dbf-b6db-4abdc5a99b6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-67dpz\" (UID: \"a847cea1-5939-4dbf-b6db-4abdc5a99b6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz" Sep 10 11:53:20.444470 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:20.444430 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a847cea1-5939-4dbf-b6db-4abdc5a99b6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-67dpz\" (UID: \"a847cea1-5939-4dbf-b6db-4abdc5a99b6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz" Sep 10 11:53:20.444623 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:53:20.444596 2582 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 10 11:53:20.444679 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:53:20.444659 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a847cea1-5939-4dbf-b6db-4abdc5a99b6e-tls-certificates podName:a847cea1-5939-4dbf-b6db-4abdc5a99b6e nodeName:}" failed. No retries permitted until 2026-09-10 11:53:20.94463998 +0000 UTC m=+108.824029391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a847cea1-5939-4dbf-b6db-4abdc5a99b6e-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-67dpz" (UID: "a847cea1-5939-4dbf-b6db-4abdc5a99b6e") : secret "prometheus-operator-admission-webhook-tls" not found Sep 10 11:53:20.948116 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:20.948082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a847cea1-5939-4dbf-b6db-4abdc5a99b6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-67dpz\" (UID: \"a847cea1-5939-4dbf-b6db-4abdc5a99b6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz" Sep 10 11:53:20.948364 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:53:20.948284 2582 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 10 11:53:20.948364 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:53:20.948358 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a847cea1-5939-4dbf-b6db-4abdc5a99b6e-tls-certificates podName:a847cea1-5939-4dbf-b6db-4abdc5a99b6e nodeName:}" failed. No retries permitted until 2026-09-10 11:53:21.948336274 +0000 UTC m=+109.827725690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a847cea1-5939-4dbf-b6db-4abdc5a99b6e-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-67dpz" (UID: "a847cea1-5939-4dbf-b6db-4abdc5a99b6e") : secret "prometheus-operator-admission-webhook-tls" not found Sep 10 11:53:21.075810 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.075778 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-586779864d-66lkv"] Sep 10 11:53:21.079909 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.079888 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.084208 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.084184 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 11:53:21.084341 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.084323 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 11:53:21.084411 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.084399 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 11:53:21.084582 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.084184 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 11:53:21.084689 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.084591 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 11:53:21.084689 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.084604 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lrm5q\"" Sep 10 11:53:21.095970 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.095808 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 11:53:21.099638 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.097984 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586779864d-66lkv"] Sep 10 11:53:21.250706 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.250622 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-oauth-config\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.250706 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.250691 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-config\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.251235 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.250742 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-serving-cert\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.251235 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.250776 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-oauth-serving-cert\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.251235 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.250829 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-service-ca\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.251235 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.250955 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-trusted-ca-bundle\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.251235 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.250994 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2p5v\" (UniqueName: \"kubernetes.io/projected/727f0789-d9da-4bd3-8bae-8c9bde049fae-kube-api-access-h2p5v\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.351790 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.351752 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-oauth-config\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.351969 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.351821 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-config\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.351969 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.351866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-serving-cert\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.351969 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.351899 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-oauth-serving-cert\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.351969 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.351924 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-service-ca\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.352209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.351995 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-trusted-ca-bundle\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.352209 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.352019 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p5v\" (UniqueName: \"kubernetes.io/projected/727f0789-d9da-4bd3-8bae-8c9bde049fae-kube-api-access-h2p5v\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.352765 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.352726 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-config\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.352882 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.352790 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-service-ca\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.352951 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.352911 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-oauth-serving-cert\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.353108 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.353075 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-trusted-ca-bundle\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.355027 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.355001 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-oauth-config\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.355449 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.355410 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-serving-cert\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.362048 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.362017 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2p5v\" (UniqueName: \"kubernetes.io/projected/727f0789-d9da-4bd3-8bae-8c9bde049fae-kube-api-access-h2p5v\") pod \"console-586779864d-66lkv\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.395902 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.395856 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:21.923295 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.923254 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586779864d-66lkv"] Sep 10 11:53:21.956930 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.956891 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a847cea1-5939-4dbf-b6db-4abdc5a99b6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-67dpz\" (UID: \"a847cea1-5939-4dbf-b6db-4abdc5a99b6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz" Sep 10 11:53:21.959839 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:21.959806 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a847cea1-5939-4dbf-b6db-4abdc5a99b6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-67dpz\" (UID: \"a847cea1-5939-4dbf-b6db-4abdc5a99b6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz" Sep 10 11:53:22.049919 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:22.049818 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k" event={"ID":"7954ff8a-86a5-484f-85c1-b03833adbc34","Type":"ContainerStarted","Data":"702c55df4baf2e340d906f26873d0e0076694b6e5164d0d29d83287915e7e7bb"} Sep 10 11:53:22.051685 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:22.051664 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz" Sep 10 11:53:22.052223 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:22.052193 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gmvjk" event={"ID":"1b5f2151-3c8b-4297-9d61-3f978d112702","Type":"ContainerStarted","Data":"5c9788f5b2653b1d20529cad13ddf3e33becfd0d313238cdb6817d1d8b5db2fe"} Sep 10 11:53:22.052334 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:22.052247 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gmvjk" event={"ID":"1b5f2151-3c8b-4297-9d61-3f978d112702","Type":"ContainerStarted","Data":"8e406794bb0e6341feba6fd550bb2dd106bd98bdd263b9d9e964223e7b7f6567"} Sep 10 11:53:22.069715 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:22.069541 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dccbb8d46-sr95k" podStartSLOduration=2.025831545 podStartE2EDuration="5.069523039s" totalImagesPullingTime="3.043691494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:22.069486788 +0000 UTC m=+109.948876217" watchObservedRunningTime="2026-09-10 11:53:22.069523039 +0000 UTC m=+109.948912474" Sep 10 11:53:22.086768 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:22.086714 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gmvjk" podStartSLOduration=1.0863567060000001 podStartE2EDuration="4.086701265s" totalImagesPullingTime="3.000344559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:22.086374918 +0000 UTC m=+109.965764351" watchObservedRunningTime="2026-09-10 11:53:22.086701265 +0000 UTC m=+109.966090697" Sep 10 11:53:22.197704 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:22.197659 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727f0789_d9da_4bd3_8bae_8c9bde049fae.slice/crio-cd6b34246771463df4ae0127b062701930b8d5b34b06d2ec779968d407c3905e WatchSource:0}: Error finding container cd6b34246771463df4ae0127b062701930b8d5b34b06d2ec779968d407c3905e: Status 404 returned error can't find the container with id cd6b34246771463df4ae0127b062701930b8d5b34b06d2ec779968d407c3905e Sep 10 11:53:22.474995 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:22.474913 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz"] Sep 10 11:53:22.478657 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:22.478627 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda847cea1_5939_4dbf_b6db_4abdc5a99b6e.slice/crio-8be3132426a83251a91ccaf114a0e298ccc1f0c9f65e6cd551d41c28468ee836 WatchSource:0}: Error finding container 8be3132426a83251a91ccaf114a0e298ccc1f0c9f65e6cd551d41c28468ee836: Status 404 returned error can't find the container with id 8be3132426a83251a91ccaf114a0e298ccc1f0c9f65e6cd551d41c28468ee836 Sep 10 11:53:23.059525 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:23.059487 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz" event={"ID":"a847cea1-5939-4dbf-b6db-4abdc5a99b6e","Type":"ContainerStarted","Data":"8be3132426a83251a91ccaf114a0e298ccc1f0c9f65e6cd551d41c28468ee836"} Sep 10 11:53:23.061689 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:23.061619 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586779864d-66lkv" event={"ID":"727f0789-d9da-4bd3-8bae-8c9bde049fae","Type":"ContainerStarted","Data":"cd6b34246771463df4ae0127b062701930b8d5b34b06d2ec779968d407c3905e"} Sep 10 11:53:23.064415 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:23.064366 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-xhwjx" event={"ID":"1be59f2c-4e40-447c-884e-f32ee3c9af46","Type":"ContainerStarted","Data":"9aa59316cfb4d7403d9fa37d3c5246a4e8e0b8bbcb81079f85e57e0a39ff1539"} Sep 10 11:53:23.082023 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:23.081968 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-xhwjx" podStartSLOduration=2.424489856 podStartE2EDuration="6.081948685s" totalImagesPullingTime="3.657458829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:23.081798621 +0000 UTC m=+110.961188053" watchObservedRunningTime="2026-09-10 11:53:23.081948685 +0000 UTC m=+110.961338117" Sep 10 11:53:24.070244 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.070135 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz" event={"ID":"a847cea1-5939-4dbf-b6db-4abdc5a99b6e","Type":"ContainerStarted","Data":"0cd68959854fd8cb834e546ae6e622cbc4fe2a7bcd1998575c70adbea1a3f6e5"} Sep 10 11:53:24.070660 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.070499 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz" Sep 10 11:53:24.077196 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.077166 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz" Sep 10 11:53:24.089957 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.089108 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-67dpz" podStartSLOduration=2.877701222 podStartE2EDuration="4.089089223s" totalImagesPullingTime="1.211388001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:24.08837258 +0000 UTC m=+111.967762014" watchObservedRunningTime="2026-09-10 11:53:24.089089223 +0000 UTC m=+111.968478657" Sep 10 11:53:24.279656 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.279607 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-74qg6"] Sep 10 11:53:24.283374 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.283349 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:24.286497 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.286365 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 10 11:53:24.286918 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.286784 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mbmj7\"" Sep 10 11:53:24.286918 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.286823 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 11:53:24.286918 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.286913 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 11:53:24.287119 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.287000 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 10 11:53:24.287247 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.287133 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 11:53:24.296530 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.296507 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-74qg6"] Sep 10 11:53:24.376169 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.376065 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5899cf32-383d-48b7-a082-c7b36ae1e145-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-74qg6\" (UID: \"5899cf32-383d-48b7-a082-c7b36ae1e145\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:24.376169 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.376156 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5899cf32-383d-48b7-a082-c7b36ae1e145-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-74qg6\" (UID: \"5899cf32-383d-48b7-a082-c7b36ae1e145\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:24.376382 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.376190 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xzps\" (UniqueName: \"kubernetes.io/projected/5899cf32-383d-48b7-a082-c7b36ae1e145-kube-api-access-5xzps\") pod \"prometheus-operator-6f4946db56-74qg6\" (UID: \"5899cf32-383d-48b7-a082-c7b36ae1e145\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:24.376382 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.376231 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5899cf32-383d-48b7-a082-c7b36ae1e145-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-74qg6\" (UID: \"5899cf32-383d-48b7-a082-c7b36ae1e145\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:24.477001 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.476967 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5899cf32-383d-48b7-a082-c7b36ae1e145-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-74qg6\" (UID: \"5899cf32-383d-48b7-a082-c7b36ae1e145\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:24.477207 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.477045 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5899cf32-383d-48b7-a082-c7b36ae1e145-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-74qg6\" (UID: \"5899cf32-383d-48b7-a082-c7b36ae1e145\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:24.477207 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.477084 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xzps\" (UniqueName: \"kubernetes.io/projected/5899cf32-383d-48b7-a082-c7b36ae1e145-kube-api-access-5xzps\") pod \"prometheus-operator-6f4946db56-74qg6\" (UID: \"5899cf32-383d-48b7-a082-c7b36ae1e145\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:24.477207 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.477126 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5899cf32-383d-48b7-a082-c7b36ae1e145-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-74qg6\" (UID: \"5899cf32-383d-48b7-a082-c7b36ae1e145\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:24.477893 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.477867 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5899cf32-383d-48b7-a082-c7b36ae1e145-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-74qg6\" (UID: \"5899cf32-383d-48b7-a082-c7b36ae1e145\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:24.482420 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.482396 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5899cf32-383d-48b7-a082-c7b36ae1e145-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-74qg6\" (UID: \"5899cf32-383d-48b7-a082-c7b36ae1e145\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:24.482568 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.482540 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5899cf32-383d-48b7-a082-c7b36ae1e145-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-74qg6\" (UID: \"5899cf32-383d-48b7-a082-c7b36ae1e145\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:24.487015 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.486993 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xzps\" (UniqueName: \"kubernetes.io/projected/5899cf32-383d-48b7-a082-c7b36ae1e145-kube-api-access-5xzps\") pod \"prometheus-operator-6f4946db56-74qg6\" (UID: \"5899cf32-383d-48b7-a082-c7b36ae1e145\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:24.597415 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:24.597379 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" Sep 10 11:53:25.021833 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:25.021799 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8927q" Sep 10 11:53:25.771944 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:25.771862 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-74qg6"] Sep 10 11:53:25.775349 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:25.775320 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5899cf32_383d_48b7_a082_c7b36ae1e145.slice/crio-00d67140e7e117e381c7aae378c250259f5043f425782173d483981590fd72ae WatchSource:0}: Error finding container 00d67140e7e117e381c7aae378c250259f5043f425782173d483981590fd72ae: Status 404 returned error can't find the container with id 00d67140e7e117e381c7aae378c250259f5043f425782173d483981590fd72ae Sep 10 11:53:26.080945 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:26.080860 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586779864d-66lkv" event={"ID":"727f0789-d9da-4bd3-8bae-8c9bde049fae","Type":"ContainerStarted","Data":"b4c95c44bbe5ffcd96f4ed87b3235256f7dd8d71e9d65b73739fec70fd5fd15f"} Sep 10 11:53:26.082202 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:26.082176 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" event={"ID":"5899cf32-383d-48b7-a082-c7b36ae1e145","Type":"ContainerStarted","Data":"00d67140e7e117e381c7aae378c250259f5043f425782173d483981590fd72ae"} Sep 10 11:53:26.113694 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:26.113650 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-586779864d-66lkv" podStartSLOduration=1.757587967 podStartE2EDuration="5.113634307s" totalImagesPullingTime="3.35604634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:26.112896821 +0000 UTC m=+113.992286253" watchObservedRunningTime="2026-09-10 11:53:26.113634307 +0000 UTC m=+113.993023738" Sep 10 11:53:28.067039 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:28.067007 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:53:28.092794 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:28.092744 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" event={"ID":"5899cf32-383d-48b7-a082-c7b36ae1e145","Type":"ContainerStarted","Data":"87edfdee02182954092c1cdb56949a32696b8996f80e9fcc057f8342c8d43c26"} Sep 10 11:53:28.092965 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:28.092810 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" event={"ID":"5899cf32-383d-48b7-a082-c7b36ae1e145","Type":"ContainerStarted","Data":"78665bd083e6bea7919f0fca0f0567e778d7e82ef304c258057f8fdb210caaf9"} Sep 10 11:53:28.112336 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:28.112291 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-74qg6" podStartSLOduration=2.789814299 podStartE2EDuration="4.112272336s" totalImagesPullingTime="1.322458037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:28.11128593 +0000 UTC m=+115.990675362" watchObservedRunningTime="2026-09-10 11:53:28.112272336 +0000 UTC m=+115.991661770" Sep 10 11:53:29.881483 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:29.881449 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-crr6h"] Sep 10 11:53:29.886287 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:29.886263 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:29.892856 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:29.892828 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kl6rx\"" Sep 10 11:53:29.893555 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:29.893530 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 11:53:29.893657 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:29.893582 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 11:53:29.893807 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:29.893785 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 11:53:30.031443 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.031415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-textfile\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.031623 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.031451 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-tls\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.031623 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.031468 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-accelerators-collector-config\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.031623 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.031553 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf6qr\" (UniqueName: \"kubernetes.io/projected/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-kube-api-access-hf6qr\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.031623 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.031594 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-root\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.031787 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.031624 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.031787 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.031700 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-metrics-client-ca\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.031787 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.031740 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-wtmp\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.031787 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.031767 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-sys\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.132770 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.132675 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf6qr\" (UniqueName: \"kubernetes.io/projected/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-kube-api-access-hf6qr\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.132770 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.132735 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-root\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.133018 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.132991 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.133073 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.133058 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-metrics-client-ca\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.133120 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.133110 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-wtmp\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.133201 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.133115 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-root\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.133201 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.133158 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-sys\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.133201 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.133189 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-textfile\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.133344 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.133215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-tls\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.133344 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.133239 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-accelerators-collector-config\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.133344 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.133268 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-sys\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.133344 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.133331 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-wtmp\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.133682 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.133654 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-textfile\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.133821 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.133793 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-metrics-client-ca\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.133870 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.133800 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-accelerators-collector-config\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.135864 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.135846 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-tls\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.135976 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.135920 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.145109 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.145082 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf6qr\" (UniqueName: \"kubernetes.io/projected/77c638dd-4da7-41bf-8e1f-d6d3d773c54c-kube-api-access-hf6qr\") pod \"node-exporter-crr6h\" (UID: \"77c638dd-4da7-41bf-8e1f-d6d3d773c54c\") " pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:30.198236 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:30.198197 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-crr6h" Sep 10 11:53:31.396784 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:31.396747 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:31.397278 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:31.396809 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:31.402330 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:31.402304 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:32.113004 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:32.112971 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-586779864d-66lkv" Sep 10 11:53:34.452890 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:34.452857 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh"] Sep 10 11:53:34.457552 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:34.457531 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" Sep 10 11:53:34.460463 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:34.460435 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 10 11:53:34.460627 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:34.460605 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cwfgd\"" Sep 10 11:53:34.469272 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:34.469217 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh"] Sep 10 11:53:34.477106 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:34.477087 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqzcd\" (UniqueName: \"kubernetes.io/projected/6b705d50-4f27-46cb-943e-94c50105943e-kube-api-access-cqzcd\") pod \"monitoring-plugin-85bbbbbf4-bc4dh\" (UID: \"6b705d50-4f27-46cb-943e-94c50105943e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" Sep 10 11:53:34.477226 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:34.477210 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b705d50-4f27-46cb-943e-94c50105943e-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-bc4dh\" (UID: \"6b705d50-4f27-46cb-943e-94c50105943e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" Sep 10 11:53:34.578445 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:34.578404 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b705d50-4f27-46cb-943e-94c50105943e-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-bc4dh\" (UID: \"6b705d50-4f27-46cb-943e-94c50105943e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" Sep 10 11:53:34.578615 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:34.578470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqzcd\" (UniqueName: \"kubernetes.io/projected/6b705d50-4f27-46cb-943e-94c50105943e-kube-api-access-cqzcd\") pod \"monitoring-plugin-85bbbbbf4-bc4dh\" (UID: \"6b705d50-4f27-46cb-943e-94c50105943e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" Sep 10 11:53:34.578615 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:53:34.578588 2582 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 10 11:53:34.578740 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:53:34.578666 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b705d50-4f27-46cb-943e-94c50105943e-monitoring-plugin-cert podName:6b705d50-4f27-46cb-943e-94c50105943e nodeName:}" failed. No retries permitted until 2026-09-10 11:53:35.078643456 +0000 UTC m=+122.958032885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/6b705d50-4f27-46cb-943e-94c50105943e-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-bc4dh" (UID: "6b705d50-4f27-46cb-943e-94c50105943e") : secret "monitoring-plugin-cert" not found Sep 10 11:53:34.588851 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:34.588819 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqzcd\" (UniqueName: \"kubernetes.io/projected/6b705d50-4f27-46cb-943e-94c50105943e-kube-api-access-cqzcd\") pod \"monitoring-plugin-85bbbbbf4-bc4dh\" (UID: \"6b705d50-4f27-46cb-943e-94c50105943e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" Sep 10 11:53:35.082954 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:35.082915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b705d50-4f27-46cb-943e-94c50105943e-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-bc4dh\" (UID: \"6b705d50-4f27-46cb-943e-94c50105943e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" Sep 10 11:53:35.085843 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:35.085819 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6b705d50-4f27-46cb-943e-94c50105943e-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-bc4dh\" (UID: \"6b705d50-4f27-46cb-943e-94c50105943e\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" Sep 10 11:53:35.368321 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:35.368248 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" Sep 10 11:53:36.220005 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:36.219938 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c638dd_4da7_41bf_8e1f_d6d3d773c54c.slice/crio-96ada807438636572a3d5339b06284cb0b45e090cc967617e2d0fb52174ae20c WatchSource:0}: Error finding container 96ada807438636572a3d5339b06284cb0b45e090cc967617e2d0fb52174ae20c: Status 404 returned error can't find the container with id 96ada807438636572a3d5339b06284cb0b45e090cc967617e2d0fb52174ae20c Sep 10 11:53:36.492273 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:36.491996 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh"] Sep 10 11:53:36.494927 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:36.494892 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b705d50_4f27_46cb_943e_94c50105943e.slice/crio-88a45d3a5fb9f78b5d5ab40a4dd4168f19dfe3e45f75c14c54e243201b5e50cf WatchSource:0}: Error finding container 88a45d3a5fb9f78b5d5ab40a4dd4168f19dfe3e45f75c14c54e243201b5e50cf: Status 404 returned error can't find the container with id 88a45d3a5fb9f78b5d5ab40a4dd4168f19dfe3e45f75c14c54e243201b5e50cf Sep 10 11:53:37.130072 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:37.130036 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-crr6h" event={"ID":"77c638dd-4da7-41bf-8e1f-d6d3d773c54c","Type":"ContainerStarted","Data":"96ada807438636572a3d5339b06284cb0b45e090cc967617e2d0fb52174ae20c"} Sep 10 11:53:37.133055 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:37.133021 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-9lg8f" event={"ID":"a88118da-c9a3-4593-8a83-5137bdd8984a","Type":"ContainerStarted","Data":"96cc14094d2da1231bb804a0752d3bee6fddd8a46b3b95abbdfd01fa32217333"} Sep 10 11:53:37.135305 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:37.135285 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-9lg8f" Sep 10 11:53:37.136453 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:37.136424 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" event={"ID":"6b705d50-4f27-46cb-943e-94c50105943e","Type":"ContainerStarted","Data":"88a45d3a5fb9f78b5d5ab40a4dd4168f19dfe3e45f75c14c54e243201b5e50cf"} Sep 10 11:53:37.147804 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:37.147779 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-9lg8f" Sep 10 11:53:37.153337 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:37.153289 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-9lg8f" podStartSLOduration=1.605978546 podStartE2EDuration="19.1532778s" totalImagesPullingTime="17.547299254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:37.15189612 +0000 UTC m=+125.031285572" watchObservedRunningTime="2026-09-10 11:53:37.1532778 +0000 UTC m=+125.032667263" Sep 10 11:53:38.144393 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:38.144346 2582 generic.go:356] "Generic (PLEG): container finished" podID="77c638dd-4da7-41bf-8e1f-d6d3d773c54c" containerID="c3c7eea05a3164e831102cafc2405ded1775c98c18317cbfa2f8e10a29575746" exitCode=0 Sep 10 11:53:38.145816 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:38.145475 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-crr6h" event={"ID":"77c638dd-4da7-41bf-8e1f-d6d3d773c54c","Type":"ContainerDied","Data":"c3c7eea05a3164e831102cafc2405ded1775c98c18317cbfa2f8e10a29575746"} Sep 10 11:53:39.150825 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:39.150728 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" event={"ID":"6b705d50-4f27-46cb-943e-94c50105943e","Type":"ContainerStarted","Data":"76f77e773804391cb2ab9bd9d4bfd24ce8689c33cc627354e06085bef14e9244"} Sep 10 11:53:39.151304 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:39.150919 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" Sep 10 11:53:39.153393 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:39.153361 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-crr6h" event={"ID":"77c638dd-4da7-41bf-8e1f-d6d3d773c54c","Type":"ContainerStarted","Data":"31d62d80f42fd10ac7e7dff574ed82ec5bdefa1c7689e03e4040b5e7907e0502"} Sep 10 11:53:39.153509 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:39.153419 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-crr6h" event={"ID":"77c638dd-4da7-41bf-8e1f-d6d3d773c54c","Type":"ContainerStarted","Data":"6e965fa572b326dd0b04218025115daa16c9641f1787e89a1c485bc6d490a789"} Sep 10 11:53:39.156787 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:39.156762 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" Sep 10 11:53:39.174931 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:39.174882 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-bc4dh" podStartSLOduration=2.779976594 podStartE2EDuration="5.174863639s" totalImagesPullingTime="2.394887045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:39.172280633 +0000 UTC m=+127.051670065" watchObservedRunningTime="2026-09-10 11:53:39.174863639 +0000 UTC m=+127.054253073" Sep 10 11:53:39.232550 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:39.232496 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-crr6h" podStartSLOduration=9.092488533 podStartE2EDuration="10.232477849s" totalImagesPullingTime="1.052120328s" totalInitContainerRuntime="87.868988ms" isStatefulPod=true podCreationTimestamp="2026-09-10 11:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:39.231909813 +0000 UTC m=+127.111299268" watchObservedRunningTime="2026-09-10 11:53:39.232477849 +0000 UTC m=+127.111867281" Sep 10 11:53:40.044082 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:40.044056 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d684479cb-4mz2c" Sep 10 11:53:41.445899 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:41.445854 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:53:41.448839 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:41.448808 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb366139-ec7c-4ae4-be68-93027b09d18c-metrics-certs\") pod \"network-metrics-daemon-lnhpb\" (UID: \"bb366139-ec7c-4ae4-be68-93027b09d18c\") " pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:53:41.645800 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:41.645762 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zrvwr\"" Sep 10 11:53:41.653799 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:41.653773 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lnhpb" Sep 10 11:53:41.969992 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:41.969959 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lnhpb"] Sep 10 11:53:41.972877 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:53:41.972852 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb366139_ec7c_4ae4_be68_93027b09d18c.slice/crio-6ac77a042cf62016c7667773c53277be120745129013370f02f8a7845ed76e3b WatchSource:0}: Error finding container 6ac77a042cf62016c7667773c53277be120745129013370f02f8a7845ed76e3b: Status 404 returned error can't find the container with id 6ac77a042cf62016c7667773c53277be120745129013370f02f8a7845ed76e3b Sep 10 11:53:42.168678 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:42.168643 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnhpb" event={"ID":"bb366139-ec7c-4ae4-be68-93027b09d18c","Type":"ContainerStarted","Data":"6ac77a042cf62016c7667773c53277be120745129013370f02f8a7845ed76e3b"} Sep 10 11:53:43.081916 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.081861 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" podUID="4258c9d2-a577-427c-b0b1-5b6f373eaef2" containerName="registry" containerID="cri-o://dd9436da02859697cde4da3c6a2421ec104fce5d15f258589330650fb2916313" gracePeriod=30 Sep 10 11:53:43.541718 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.541694 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:53:43.665552 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.665519 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4258c9d2-a577-427c-b0b1-5b6f373eaef2-installation-pull-secrets\") pod \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " Sep 10 11:53:43.665709 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.665576 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-bound-sa-token\") pod \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " Sep 10 11:53:43.666697 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.666020 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4258c9d2-a577-427c-b0b1-5b6f373eaef2-image-registry-private-configuration\") pod \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " Sep 10 11:53:43.666697 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.666088 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls\") pod \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " Sep 10 11:53:43.666697 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.666136 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4258c9d2-a577-427c-b0b1-5b6f373eaef2-trusted-ca\") pod \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " Sep 10 11:53:43.666697 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.666284 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-certificates\") pod \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " Sep 10 11:53:43.666697 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.666404 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gcqs\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-kube-api-access-2gcqs\") pod \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " Sep 10 11:53:43.666697 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.666435 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4258c9d2-a577-427c-b0b1-5b6f373eaef2-ca-trust-extracted\") pod \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\" (UID: \"4258c9d2-a577-427c-b0b1-5b6f373eaef2\") " Sep 10 11:53:43.667324 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.667296 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4258c9d2-a577-427c-b0b1-5b6f373eaef2" (UID: "4258c9d2-a577-427c-b0b1-5b6f373eaef2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 11:53:43.669252 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.667967 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4258c9d2-a577-427c-b0b1-5b6f373eaef2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4258c9d2-a577-427c-b0b1-5b6f373eaef2" (UID: "4258c9d2-a577-427c-b0b1-5b6f373eaef2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 11:53:43.669252 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.668393 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4258c9d2-a577-427c-b0b1-5b6f373eaef2" (UID: "4258c9d2-a577-427c-b0b1-5b6f373eaef2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 11:53:43.670173 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.669669 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4258c9d2-a577-427c-b0b1-5b6f373eaef2" (UID: "4258c9d2-a577-427c-b0b1-5b6f373eaef2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 11:53:43.670390 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.670348 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-kube-api-access-2gcqs" (OuterVolumeSpecName: "kube-api-access-2gcqs") pod "4258c9d2-a577-427c-b0b1-5b6f373eaef2" (UID: "4258c9d2-a577-427c-b0b1-5b6f373eaef2"). InnerVolumeSpecName "kube-api-access-2gcqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 11:53:43.670968 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.670935 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4258c9d2-a577-427c-b0b1-5b6f373eaef2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4258c9d2-a577-427c-b0b1-5b6f373eaef2" (UID: "4258c9d2-a577-427c-b0b1-5b6f373eaef2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 11:53:43.677463 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.677440 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4258c9d2-a577-427c-b0b1-5b6f373eaef2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4258c9d2-a577-427c-b0b1-5b6f373eaef2" (UID: "4258c9d2-a577-427c-b0b1-5b6f373eaef2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 11:53:43.678774 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.678751 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4258c9d2-a577-427c-b0b1-5b6f373eaef2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4258c9d2-a577-427c-b0b1-5b6f373eaef2" (UID: "4258c9d2-a577-427c-b0b1-5b6f373eaef2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 11:53:43.769051 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.768963 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4258c9d2-a577-427c-b0b1-5b6f373eaef2-ca-trust-extracted\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:53:43.769051 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.769001 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4258c9d2-a577-427c-b0b1-5b6f373eaef2-installation-pull-secrets\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:53:43.769051 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.769017 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-bound-sa-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:53:43.769051 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.769036 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4258c9d2-a577-427c-b0b1-5b6f373eaef2-image-registry-private-configuration\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:53:43.769051 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.769049 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-tls\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:53:43.769424 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.769063 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4258c9d2-a577-427c-b0b1-5b6f373eaef2-trusted-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:53:43.769424 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.769075 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4258c9d2-a577-427c-b0b1-5b6f373eaef2-registry-certificates\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:53:43.769424 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:43.769087 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gcqs\" (UniqueName: \"kubernetes.io/projected/4258c9d2-a577-427c-b0b1-5b6f373eaef2-kube-api-access-2gcqs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:53:44.180248 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:44.180216 2582 generic.go:356] "Generic (PLEG): container finished" podID="4258c9d2-a577-427c-b0b1-5b6f373eaef2" containerID="dd9436da02859697cde4da3c6a2421ec104fce5d15f258589330650fb2916313" exitCode=0 Sep 10 11:53:44.180830 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:44.180803 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" event={"ID":"4258c9d2-a577-427c-b0b1-5b6f373eaef2","Type":"ContainerDied","Data":"dd9436da02859697cde4da3c6a2421ec104fce5d15f258589330650fb2916313"} Sep 10 11:53:44.181003 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:44.180977 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" event={"ID":"4258c9d2-a577-427c-b0b1-5b6f373eaef2","Type":"ContainerDied","Data":"e57ab2ab5aeec0f2b51bc8b251f7c5777437ad77a10846d6bb120afb5abdcd56"} Sep 10 11:53:44.181303 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:44.181281 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d8fcd69-6lg7t" Sep 10 11:53:44.186207 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:44.186181 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnhpb" event={"ID":"bb366139-ec7c-4ae4-be68-93027b09d18c","Type":"ContainerStarted","Data":"47415b4ee2ea74fa18057bf9695b9651d756d9c3513e42578585de2d92c02474"} Sep 10 11:53:44.186294 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:44.186229 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lnhpb" event={"ID":"bb366139-ec7c-4ae4-be68-93027b09d18c","Type":"ContainerStarted","Data":"de66fc4da4a85853a62f264925f0caf688f5b69625f870fc9bac0fe10edc0d4f"} Sep 10 11:53:44.186918 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:44.186895 2582 scope.go:117] "RemoveContainer" containerID="dd9436da02859697cde4da3c6a2421ec104fce5d15f258589330650fb2916313" Sep 10 11:53:44.196937 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:44.196915 2582 scope.go:117] "RemoveContainer" containerID="dd9436da02859697cde4da3c6a2421ec104fce5d15f258589330650fb2916313" Sep 10 11:53:44.197289 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:53:44.197264 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9436da02859697cde4da3c6a2421ec104fce5d15f258589330650fb2916313\": container with ID starting with dd9436da02859697cde4da3c6a2421ec104fce5d15f258589330650fb2916313 not found: ID does not exist" containerID="dd9436da02859697cde4da3c6a2421ec104fce5d15f258589330650fb2916313" Sep 10 11:53:44.197376 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:44.197298 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9436da02859697cde4da3c6a2421ec104fce5d15f258589330650fb2916313"} err="failed to get container status \"dd9436da02859697cde4da3c6a2421ec104fce5d15f258589330650fb2916313\": rpc error: code = NotFound desc = could not find container \"dd9436da02859697cde4da3c6a2421ec104fce5d15f258589330650fb2916313\": container with ID starting with dd9436da02859697cde4da3c6a2421ec104fce5d15f258589330650fb2916313 not found: ID does not exist" Sep 10 11:53:44.205276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:44.203574 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lnhpb" podStartSLOduration=129.650898425 podStartE2EDuration="2m11.203558303s" totalImagesPullingTime="1.552659878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:44.203030335 +0000 UTC m=+132.082419769" watchObservedRunningTime="2026-09-10 11:53:44.203558303 +0000 UTC m=+132.082947739" Sep 10 11:53:44.220001 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:44.219973 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d8fcd69-6lg7t"] Sep 10 11:53:44.222737 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:44.222715 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d8fcd69-6lg7t"] Sep 10 11:53:44.629571 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:44.629538 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4258c9d2-a577-427c-b0b1-5b6f373eaef2" path="/var/lib/kubelet/pods/4258c9d2-a577-427c-b0b1-5b6f373eaef2/volumes" Sep 10 11:53:46.544183 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:53:46.541325 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586779864d-66lkv"] Sep 10 11:54:11.564329 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.564273 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-586779864d-66lkv" podUID="727f0789-d9da-4bd3-8bae-8c9bde049fae" containerName="console" containerID="cri-o://b4c95c44bbe5ffcd96f4ed87b3235256f7dd8d71e9d65b73739fec70fd5fd15f" gracePeriod=15 Sep 10 11:54:11.832335 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.832275 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586779864d-66lkv" Sep 10 11:54:11.990157 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.990107 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-trusted-ca-bundle\") pod \"727f0789-d9da-4bd3-8bae-8c9bde049fae\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " Sep 10 11:54:11.990320 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.990202 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-service-ca\") pod \"727f0789-d9da-4bd3-8bae-8c9bde049fae\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " Sep 10 11:54:11.990320 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.990233 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-oauth-config\") pod \"727f0789-d9da-4bd3-8bae-8c9bde049fae\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " Sep 10 11:54:11.990320 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.990269 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-config\") pod \"727f0789-d9da-4bd3-8bae-8c9bde049fae\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " Sep 10 11:54:11.990320 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.990294 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-oauth-serving-cert\") pod \"727f0789-d9da-4bd3-8bae-8c9bde049fae\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " Sep 10 11:54:11.990320 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.990316 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2p5v\" (UniqueName: \"kubernetes.io/projected/727f0789-d9da-4bd3-8bae-8c9bde049fae-kube-api-access-h2p5v\") pod \"727f0789-d9da-4bd3-8bae-8c9bde049fae\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " Sep 10 11:54:11.990560 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.990452 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-serving-cert\") pod \"727f0789-d9da-4bd3-8bae-8c9bde049fae\" (UID: \"727f0789-d9da-4bd3-8bae-8c9bde049fae\") " Sep 10 11:54:11.990676 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.990648 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "727f0789-d9da-4bd3-8bae-8c9bde049fae" (UID: "727f0789-d9da-4bd3-8bae-8c9bde049fae"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 11:54:11.990747 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.990685 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-service-ca" (OuterVolumeSpecName: "service-ca") pod "727f0789-d9da-4bd3-8bae-8c9bde049fae" (UID: "727f0789-d9da-4bd3-8bae-8c9bde049fae"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 11:54:11.990747 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.990692 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "727f0789-d9da-4bd3-8bae-8c9bde049fae" (UID: "727f0789-d9da-4bd3-8bae-8c9bde049fae"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 11:54:11.990836 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.990751 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-config" (OuterVolumeSpecName: "console-config") pod "727f0789-d9da-4bd3-8bae-8c9bde049fae" (UID: "727f0789-d9da-4bd3-8bae-8c9bde049fae"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 11:54:11.992676 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.992654 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "727f0789-d9da-4bd3-8bae-8c9bde049fae" (UID: "727f0789-d9da-4bd3-8bae-8c9bde049fae"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 11:54:11.992752 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.992672 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "727f0789-d9da-4bd3-8bae-8c9bde049fae" (UID: "727f0789-d9da-4bd3-8bae-8c9bde049fae"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 11:54:11.992752 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:11.992685 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727f0789-d9da-4bd3-8bae-8c9bde049fae-kube-api-access-h2p5v" (OuterVolumeSpecName: "kube-api-access-h2p5v") pod "727f0789-d9da-4bd3-8bae-8c9bde049fae" (UID: "727f0789-d9da-4bd3-8bae-8c9bde049fae"). InnerVolumeSpecName "kube-api-access-h2p5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 11:54:12.092096 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.092021 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:54:12.092096 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.092054 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-oauth-config\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:54:12.092096 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.092068 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-config\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:54:12.092096 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.092081 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-oauth-serving-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:54:12.092096 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.092094 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2p5v\" (UniqueName: \"kubernetes.io/projected/727f0789-d9da-4bd3-8bae-8c9bde049fae-kube-api-access-h2p5v\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:54:12.092368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.092107 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/727f0789-d9da-4bd3-8bae-8c9bde049fae-console-serving-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:54:12.092368 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.092121 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/727f0789-d9da-4bd3-8bae-8c9bde049fae-trusted-ca-bundle\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:54:12.286295 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.286254 2582 generic.go:356] "Generic (PLEG): container finished" podID="727f0789-d9da-4bd3-8bae-8c9bde049fae" containerID="b4c95c44bbe5ffcd96f4ed87b3235256f7dd8d71e9d65b73739fec70fd5fd15f" exitCode=0 Sep 10 11:54:12.286464 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.286309 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586779864d-66lkv" Sep 10 11:54:12.286464 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.286338 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586779864d-66lkv" event={"ID":"727f0789-d9da-4bd3-8bae-8c9bde049fae","Type":"ContainerDied","Data":"b4c95c44bbe5ffcd96f4ed87b3235256f7dd8d71e9d65b73739fec70fd5fd15f"} Sep 10 11:54:12.286464 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.286386 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586779864d-66lkv" event={"ID":"727f0789-d9da-4bd3-8bae-8c9bde049fae","Type":"ContainerDied","Data":"cd6b34246771463df4ae0127b062701930b8d5b34b06d2ec779968d407c3905e"} Sep 10 11:54:12.286464 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.286406 2582 scope.go:117] "RemoveContainer" containerID="b4c95c44bbe5ffcd96f4ed87b3235256f7dd8d71e9d65b73739fec70fd5fd15f" Sep 10 11:54:12.295275 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.295256 2582 scope.go:117] "RemoveContainer" containerID="b4c95c44bbe5ffcd96f4ed87b3235256f7dd8d71e9d65b73739fec70fd5fd15f" Sep 10 11:54:12.295518 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:54:12.295501 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c95c44bbe5ffcd96f4ed87b3235256f7dd8d71e9d65b73739fec70fd5fd15f\": container with ID starting with b4c95c44bbe5ffcd96f4ed87b3235256f7dd8d71e9d65b73739fec70fd5fd15f not found: ID does not exist" containerID="b4c95c44bbe5ffcd96f4ed87b3235256f7dd8d71e9d65b73739fec70fd5fd15f" Sep 10 11:54:12.295583 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.295525 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c95c44bbe5ffcd96f4ed87b3235256f7dd8d71e9d65b73739fec70fd5fd15f"} err="failed to get container status \"b4c95c44bbe5ffcd96f4ed87b3235256f7dd8d71e9d65b73739fec70fd5fd15f\": rpc error: code = NotFound desc = could not find container \"b4c95c44bbe5ffcd96f4ed87b3235256f7dd8d71e9d65b73739fec70fd5fd15f\": container with ID starting with b4c95c44bbe5ffcd96f4ed87b3235256f7dd8d71e9d65b73739fec70fd5fd15f not found: ID does not exist" Sep 10 11:54:12.309247 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.309224 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586779864d-66lkv"] Sep 10 11:54:12.315478 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.315458 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-586779864d-66lkv"] Sep 10 11:54:12.627961 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:12.627922 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="727f0789-d9da-4bd3-8bae-8c9bde049fae" path="/var/lib/kubelet/pods/727f0789-d9da-4bd3-8bae-8c9bde049fae/volumes" Sep 10 11:54:15.301416 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:15.301387 2582 generic.go:356] "Generic (PLEG): container finished" podID="313f87bb-5d68-4539-a59b-226c262d5309" containerID="87143ecf10a4c8351e66c98f40624a07dad604c454d271816176454230e9e69f" exitCode=0 Sep 10 11:54:15.301811 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:15.301441 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" event={"ID":"313f87bb-5d68-4539-a59b-226c262d5309","Type":"ContainerDied","Data":"87143ecf10a4c8351e66c98f40624a07dad604c454d271816176454230e9e69f"} Sep 10 11:54:15.301811 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:15.301768 2582 scope.go:117] "RemoveContainer" containerID="87143ecf10a4c8351e66c98f40624a07dad604c454d271816176454230e9e69f" Sep 10 11:54:16.306580 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:16.306544 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-kwhz7" event={"ID":"313f87bb-5d68-4539-a59b-226c262d5309","Type":"ContainerStarted","Data":"ed8d4349ba017e151e3c958fa0dd4386fd9ea64e5d0e5ab1791b7a0b92f62344"} Sep 10 11:54:54.152882 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.152841 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-776865cb97-cc5bx"] Sep 10 11:54:54.153499 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.153245 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4258c9d2-a577-427c-b0b1-5b6f373eaef2" containerName="registry" Sep 10 11:54:54.153499 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.153261 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4258c9d2-a577-427c-b0b1-5b6f373eaef2" containerName="registry" Sep 10 11:54:54.153499 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.153288 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727f0789-d9da-4bd3-8bae-8c9bde049fae" containerName="console" Sep 10 11:54:54.153499 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.153296 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="727f0789-d9da-4bd3-8bae-8c9bde049fae" containerName="console" Sep 10 11:54:54.153499 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.153361 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4258c9d2-a577-427c-b0b1-5b6f373eaef2" containerName="registry" Sep 10 11:54:54.153499 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.153372 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="727f0789-d9da-4bd3-8bae-8c9bde049fae" containerName="console" Sep 10 11:54:54.155427 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.155406 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.158587 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.158564 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 10 11:54:54.158787 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.158770 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 10 11:54:54.158852 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.158811 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7t6qq\"" Sep 10 11:54:54.158900 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.158776 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 10 11:54:54.158945 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.158927 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 10 11:54:54.159463 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.159446 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 10 11:54:54.166415 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.166393 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 10 11:54:54.178688 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.178659 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-776865cb97-cc5bx"] Sep 10 11:54:54.200453 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.200417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qwb2\" (UniqueName: \"kubernetes.io/projected/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-kube-api-access-2qwb2\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.200614 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.200489 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.200614 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.200513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.200614 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.200571 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-serving-certs-ca-bundle\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.200614 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.200601 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-metrics-client-ca\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.200762 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.200647 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-federate-client-tls\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.200762 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.200678 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-telemeter-client-tls\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.200762 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.200704 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-secret-telemeter-client\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.301463 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.301420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.301463 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.301469 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.301700 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.301501 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-serving-certs-ca-bundle\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.301700 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.301518 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-metrics-client-ca\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.301700 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.301549 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-federate-client-tls\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.301700 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.301593 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-telemeter-client-tls\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.301700 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.301621 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-secret-telemeter-client\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.301700 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.301647 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qwb2\" (UniqueName: \"kubernetes.io/projected/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-kube-api-access-2qwb2\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.302329 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.302303 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-serving-certs-ca-bundle\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.302514 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.302496 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.302832 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.302789 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-metrics-client-ca\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.304570 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.304541 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-secret-telemeter-client\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.304661 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.304589 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-federate-client-tls\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.304661 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.304652 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-telemeter-client-tls\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.304745 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.304722 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.311763 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.311738 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qwb2\" (UniqueName: \"kubernetes.io/projected/bf999d19-c4f2-472c-b8ec-bb1ac70195d4-kube-api-access-2qwb2\") pod \"telemeter-client-776865cb97-cc5bx\" (UID: \"bf999d19-c4f2-472c-b8ec-bb1ac70195d4\") " pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.464439 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.464412 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" Sep 10 11:54:54.745467 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:54.745439 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-776865cb97-cc5bx"] Sep 10 11:54:54.747218 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:54:54.747186 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf999d19_c4f2_472c_b8ec_bb1ac70195d4.slice/crio-b32f5e525217e84fdfb5a5b5fd6c5b210cd5969bd3fc6b05d71d8e6dc2bbfabb WatchSource:0}: Error finding container b32f5e525217e84fdfb5a5b5fd6c5b210cd5969bd3fc6b05d71d8e6dc2bbfabb: Status 404 returned error can't find the container with id b32f5e525217e84fdfb5a5b5fd6c5b210cd5969bd3fc6b05d71d8e6dc2bbfabb Sep 10 11:54:55.445482 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:55.445448 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" event={"ID":"bf999d19-c4f2-472c-b8ec-bb1ac70195d4","Type":"ContainerStarted","Data":"b32f5e525217e84fdfb5a5b5fd6c5b210cd5969bd3fc6b05d71d8e6dc2bbfabb"} Sep 10 11:54:57.455610 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:57.455571 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" event={"ID":"bf999d19-c4f2-472c-b8ec-bb1ac70195d4","Type":"ContainerStarted","Data":"0c9622b33314d56cbfe5418d0927d8329cb98854b4534359b7c8c972766e31c0"} Sep 10 11:54:58.461023 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:58.460989 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" event={"ID":"bf999d19-c4f2-472c-b8ec-bb1ac70195d4","Type":"ContainerStarted","Data":"0771cf1ff3273c3c44bd79bafd7e00e1ff9bbcda3024e869f468b961e58f16d1"} Sep 10 11:54:58.461432 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:58.461043 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" event={"ID":"bf999d19-c4f2-472c-b8ec-bb1ac70195d4","Type":"ContainerStarted","Data":"96b446d3340b995559416bf84b70165630a4687b32f1a5a209804ac3d3885b73"} Sep 10 11:54:58.486876 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:54:58.486825 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-776865cb97-cc5bx" podStartSLOduration=1.546058769 podStartE2EDuration="4.486806305s" totalImagesPullingTime="2.940747536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:54:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:54:58.485388758 +0000 UTC m=+206.364778190" watchObservedRunningTime="2026-09-10 11:54:58.486806305 +0000 UTC m=+206.366195738" Sep 10 11:55:07.368745 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.368711 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b6766bf8d-4c2rg"] Sep 10 11:55:07.370758 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.370743 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.373995 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.373973 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 11:55:07.374122 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.374106 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 11:55:07.374683 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.374664 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 11:55:07.374767 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.374714 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lrm5q\"" Sep 10 11:55:07.374767 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.374731 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 11:55:07.374929 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.374913 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 11:55:07.379858 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.379835 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 11:55:07.385959 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.385935 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b6766bf8d-4c2rg"] Sep 10 11:55:07.400595 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.400565 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-console-config\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.400694 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.400603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-oauth-serving-cert\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.400694 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.400663 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/611baf38-a738-4227-b569-e1d6183d2b45-console-oauth-config\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.400805 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.400692 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-trusted-ca-bundle\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.400805 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.400722 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-service-ca\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.400805 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.400751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjl2d\" (UniqueName: \"kubernetes.io/projected/611baf38-a738-4227-b569-e1d6183d2b45-kube-api-access-hjl2d\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.400945 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.400844 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/611baf38-a738-4227-b569-e1d6183d2b45-console-serving-cert\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.501779 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.501746 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/611baf38-a738-4227-b569-e1d6183d2b45-console-serving-cert\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.501939 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.501798 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-console-config\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.501939 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.501826 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-oauth-serving-cert\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.501939 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.501857 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/611baf38-a738-4227-b569-e1d6183d2b45-console-oauth-config\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.501939 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.501879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-trusted-ca-bundle\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.501939 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.501903 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-service-ca\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.502239 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.501984 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjl2d\" (UniqueName: \"kubernetes.io/projected/611baf38-a738-4227-b569-e1d6183d2b45-kube-api-access-hjl2d\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.502672 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.502647 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-service-ca\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.502765 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.502703 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-console-config\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.502877 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.502861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-trusted-ca-bundle\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.502940 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.502917 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-oauth-serving-cert\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.504452 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.504426 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/611baf38-a738-4227-b569-e1d6183d2b45-console-serving-cert\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.504900 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.504876 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/611baf38-a738-4227-b569-e1d6183d2b45-console-oauth-config\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.510956 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.510934 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjl2d\" (UniqueName: \"kubernetes.io/projected/611baf38-a738-4227-b569-e1d6183d2b45-kube-api-access-hjl2d\") pod \"console-b6766bf8d-4c2rg\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.679998 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.679890 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:07.949884 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:07.949851 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b6766bf8d-4c2rg"] Sep 10 11:55:07.952805 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:55:07.952781 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611baf38_a738_4227_b569_e1d6183d2b45.slice/crio-c9382f8f200770c927b095928d378e9295f51f0941e57217209219555effffd1 WatchSource:0}: Error finding container c9382f8f200770c927b095928d378e9295f51f0941e57217209219555effffd1: Status 404 returned error can't find the container with id c9382f8f200770c927b095928d378e9295f51f0941e57217209219555effffd1 Sep 10 11:55:08.498834 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:08.498797 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6766bf8d-4c2rg" event={"ID":"611baf38-a738-4227-b569-e1d6183d2b45","Type":"ContainerStarted","Data":"fc3b850fa9cfca0bb8fa15a41fccaad0c9e997df624737f3b822afb1daa40bf9"} Sep 10 11:55:08.499234 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:08.498854 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6766bf8d-4c2rg" event={"ID":"611baf38-a738-4227-b569-e1d6183d2b45","Type":"ContainerStarted","Data":"c9382f8f200770c927b095928d378e9295f51f0941e57217209219555effffd1"} Sep 10 11:55:08.518652 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:08.518603 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b6766bf8d-4c2rg" podStartSLOduration=1.518586982 podStartE2EDuration="1.518586982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:55:08.517905895 +0000 UTC m=+216.397295327" watchObservedRunningTime="2026-09-10 11:55:08.518586982 +0000 UTC m=+216.397976414" Sep 10 11:55:17.681073 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:17.681032 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:17.681504 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:17.681093 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:17.685754 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:17.685731 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:55:18.541374 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:55:18.541349 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:56:32.532940 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:32.532911 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 11:56:32.545531 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:32.545506 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 11:56:32.546686 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:32.546666 2582 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 11:56:46.376216 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.376172 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-67lkx"] Sep 10 11:56:46.379320 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.379305 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-67lkx" Sep 10 11:56:46.384628 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.384600 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 11:56:46.384753 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.384692 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 11:56:46.385679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.385657 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-czc5p\"" Sep 10 11:56:46.390936 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.390619 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-67lkx"] Sep 10 11:56:46.455222 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.455191 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5mhn\" (UniqueName: \"kubernetes.io/projected/15f98612-dee0-417e-89f7-56485210ad6c-kube-api-access-j5mhn\") pod \"cert-manager-55b4d954d5-67lkx\" (UID: \"15f98612-dee0-417e-89f7-56485210ad6c\") " pod="cert-manager/cert-manager-55b4d954d5-67lkx" Sep 10 11:56:46.455354 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.455305 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15f98612-dee0-417e-89f7-56485210ad6c-bound-sa-token\") pod \"cert-manager-55b4d954d5-67lkx\" (UID: \"15f98612-dee0-417e-89f7-56485210ad6c\") " pod="cert-manager/cert-manager-55b4d954d5-67lkx" Sep 10 11:56:46.556218 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.556133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15f98612-dee0-417e-89f7-56485210ad6c-bound-sa-token\") pod \"cert-manager-55b4d954d5-67lkx\" (UID: \"15f98612-dee0-417e-89f7-56485210ad6c\") " pod="cert-manager/cert-manager-55b4d954d5-67lkx" Sep 10 11:56:46.556402 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.556250 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mhn\" (UniqueName: \"kubernetes.io/projected/15f98612-dee0-417e-89f7-56485210ad6c-kube-api-access-j5mhn\") pod \"cert-manager-55b4d954d5-67lkx\" (UID: \"15f98612-dee0-417e-89f7-56485210ad6c\") " pod="cert-manager/cert-manager-55b4d954d5-67lkx" Sep 10 11:56:46.565410 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.565384 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5mhn\" (UniqueName: \"kubernetes.io/projected/15f98612-dee0-417e-89f7-56485210ad6c-kube-api-access-j5mhn\") pod \"cert-manager-55b4d954d5-67lkx\" (UID: \"15f98612-dee0-417e-89f7-56485210ad6c\") " pod="cert-manager/cert-manager-55b4d954d5-67lkx" Sep 10 11:56:46.565410 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.565396 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/15f98612-dee0-417e-89f7-56485210ad6c-bound-sa-token\") pod \"cert-manager-55b4d954d5-67lkx\" (UID: \"15f98612-dee0-417e-89f7-56485210ad6c\") " pod="cert-manager/cert-manager-55b4d954d5-67lkx" Sep 10 11:56:46.693155 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.693126 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-67lkx" Sep 10 11:56:46.967970 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.967896 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-67lkx"] Sep 10 11:56:46.970906 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:56:46.970877 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15f98612_dee0_417e_89f7_56485210ad6c.slice/crio-f9ab66290789373355a2b0550e109e408e9e57aa8fcc7ac785a31287cd86de4a WatchSource:0}: Error finding container f9ab66290789373355a2b0550e109e408e9e57aa8fcc7ac785a31287cd86de4a: Status 404 returned error can't find the container with id f9ab66290789373355a2b0550e109e408e9e57aa8fcc7ac785a31287cd86de4a Sep 10 11:56:46.972648 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:46.972631 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 11:56:47.875215 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:47.875180 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-67lkx" event={"ID":"15f98612-dee0-417e-89f7-56485210ad6c","Type":"ContainerStarted","Data":"f9ab66290789373355a2b0550e109e408e9e57aa8fcc7ac785a31287cd86de4a"} Sep 10 11:56:49.936953 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:49.936926 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq"] Sep 10 11:56:49.940685 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:49.940665 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq" Sep 10 11:56:49.945779 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:49.945761 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jxjf9\"" Sep 10 11:56:49.951315 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:49.950088 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 11:56:49.951315 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:49.950389 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 11:56:49.957120 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:49.957098 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq"] Sep 10 11:56:50.088189 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:50.088165 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf2tm\" (UniqueName: \"kubernetes.io/projected/0c05e6c3-2511-42f7-94c9-9dd4cae04099-kube-api-access-rf2tm\") pod \"openshift-lws-operator-fd8ccff4c-sxxvq\" (UID: \"0c05e6c3-2511-42f7-94c9-9dd4cae04099\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq" Sep 10 11:56:50.088279 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:50.088208 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c05e6c3-2511-42f7-94c9-9dd4cae04099-tmp\") pod \"openshift-lws-operator-fd8ccff4c-sxxvq\" (UID: \"0c05e6c3-2511-42f7-94c9-9dd4cae04099\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq" Sep 10 11:56:50.189500 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:50.189419 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf2tm\" (UniqueName: \"kubernetes.io/projected/0c05e6c3-2511-42f7-94c9-9dd4cae04099-kube-api-access-rf2tm\") pod \"openshift-lws-operator-fd8ccff4c-sxxvq\" (UID: \"0c05e6c3-2511-42f7-94c9-9dd4cae04099\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq" Sep 10 11:56:50.189500 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:50.189462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c05e6c3-2511-42f7-94c9-9dd4cae04099-tmp\") pod \"openshift-lws-operator-fd8ccff4c-sxxvq\" (UID: \"0c05e6c3-2511-42f7-94c9-9dd4cae04099\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq" Sep 10 11:56:50.189820 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:50.189800 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c05e6c3-2511-42f7-94c9-9dd4cae04099-tmp\") pod \"openshift-lws-operator-fd8ccff4c-sxxvq\" (UID: \"0c05e6c3-2511-42f7-94c9-9dd4cae04099\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq" Sep 10 11:56:50.203134 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:50.203109 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf2tm\" (UniqueName: \"kubernetes.io/projected/0c05e6c3-2511-42f7-94c9-9dd4cae04099-kube-api-access-rf2tm\") pod \"openshift-lws-operator-fd8ccff4c-sxxvq\" (UID: \"0c05e6c3-2511-42f7-94c9-9dd4cae04099\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq" Sep 10 11:56:50.252065 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:50.252043 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq" Sep 10 11:56:50.531662 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:50.531639 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq"] Sep 10 11:56:50.534209 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:56:50.534183 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c05e6c3_2511_42f7_94c9_9dd4cae04099.slice/crio-73158c07d6bfe04d72047cfb53dc381f95c39c2c8f840935952c27ff4a93e715 WatchSource:0}: Error finding container 73158c07d6bfe04d72047cfb53dc381f95c39c2c8f840935952c27ff4a93e715: Status 404 returned error can't find the container with id 73158c07d6bfe04d72047cfb53dc381f95c39c2c8f840935952c27ff4a93e715 Sep 10 11:56:50.892213 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:50.892097 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-67lkx" event={"ID":"15f98612-dee0-417e-89f7-56485210ad6c","Type":"ContainerStarted","Data":"aff4f377666ee1e5d2bccca786b299f7cb2f202f88aa8bebacb0a3ae7015167f"} Sep 10 11:56:50.893428 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:50.893404 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq" event={"ID":"0c05e6c3-2511-42f7-94c9-9dd4cae04099","Type":"ContainerStarted","Data":"73158c07d6bfe04d72047cfb53dc381f95c39c2c8f840935952c27ff4a93e715"} Sep 10 11:56:50.924828 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:50.924767 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-67lkx" podStartSLOduration=1.854289073 podStartE2EDuration="4.924749204s" totalImagesPullingTime="3.070460131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:56:50.922036206 +0000 UTC m=+318.801425637" watchObservedRunningTime="2026-09-10 11:56:50.924749204 +0000 UTC m=+318.804138637" Sep 10 11:56:53.906440 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:53.906399 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq" event={"ID":"0c05e6c3-2511-42f7-94c9-9dd4cae04099","Type":"ContainerStarted","Data":"092a6872ddf4890015316a72baf8fc58e49f8c537c300f2cdbbbe69f383c5a14"} Sep 10 11:56:53.924096 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:56:53.924040 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sxxvq" podStartSLOduration=2.569722199 podStartE2EDuration="4.924022732s" totalImagesPullingTime="2.354300533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:56:53.923798132 +0000 UTC m=+321.803187567" watchObservedRunningTime="2026-09-10 11:56:53.924022732 +0000 UTC m=+321.803412165" Sep 10 11:57:13.086807 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.086767 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx"] Sep 10 11:57:13.154788 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.154756 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx"] Sep 10 11:57:13.154947 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.154880 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.162996 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.162976 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 11:57:13.163109 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.163025 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-64jsz\"" Sep 10 11:57:13.163440 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.163424 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 11:57:13.163776 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.163761 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 11:57:13.164227 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.164211 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 11:57:13.174964 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.174939 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-webhook-cert\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.175045 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.174995 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.175045 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.175040 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-manifests\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.175118 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.175060 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-tmp\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.175172 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.175119 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kkgk\" (UniqueName: \"kubernetes.io/projected/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-kube-api-access-8kkgk\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.276064 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.276031 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.276064 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.276067 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-manifests\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.276346 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.276083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-tmp\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.276346 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.276106 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kkgk\" (UniqueName: \"kubernetes.io/projected/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-kube-api-access-8kkgk\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.276346 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.276210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-webhook-cert\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.276528 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.276504 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-manifests\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.278688 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.278660 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-tmp\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.278800 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.278741 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-webhook-cert\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.278863 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.278814 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.289565 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.289539 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kkgk\" (UniqueName: \"kubernetes.io/projected/649bbbfd-c48f-4a44-b7bc-e0fbb0815fee-kube-api-access-8kkgk\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-58rlx\" (UID: \"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.464985 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.464947 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:13.758068 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.758046 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx"] Sep 10 11:57:13.762607 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:57:13.762574 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649bbbfd_c48f_4a44_b7bc_e0fbb0815fee.slice/crio-76c2d892fca7216ab2f8c06ac0005b4f9c8bc357c5fbfc9e1aa11b273b7f6345 WatchSource:0}: Error finding container 76c2d892fca7216ab2f8c06ac0005b4f9c8bc357c5fbfc9e1aa11b273b7f6345: Status 404 returned error can't find the container with id 76c2d892fca7216ab2f8c06ac0005b4f9c8bc357c5fbfc9e1aa11b273b7f6345 Sep 10 11:57:13.981499 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:13.981462 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" event={"ID":"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee","Type":"ContainerStarted","Data":"76c2d892fca7216ab2f8c06ac0005b4f9c8bc357c5fbfc9e1aa11b273b7f6345"} Sep 10 11:57:16.998887 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:16.998800 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" event={"ID":"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee","Type":"ContainerStarted","Data":"56044e1836e4a2079e1071eb3d72b6c35a4153e4db53c63b42516663b5b83d54"} Sep 10 11:57:18.004688 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:18.004647 2582 generic.go:356] "Generic (PLEG): container finished" podID="649bbbfd-c48f-4a44-b7bc-e0fbb0815fee" containerID="56044e1836e4a2079e1071eb3d72b6c35a4153e4db53c63b42516663b5b83d54" exitCode=0 Sep 10 11:57:18.005039 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:18.004706 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" event={"ID":"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee","Type":"ContainerDied","Data":"56044e1836e4a2079e1071eb3d72b6c35a4153e4db53c63b42516663b5b83d54"} Sep 10 11:57:19.012543 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:19.012498 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" event={"ID":"649bbbfd-c48f-4a44-b7bc-e0fbb0815fee","Type":"ContainerStarted","Data":"6bfd52019cadee1ca5f61bedfe09f6480a667e2394a5e4231329c822e0e41124"} Sep 10 11:57:19.012918 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:19.012618 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:19.046822 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:19.046770 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" podStartSLOduration=1.4930879209999999 podStartE2EDuration="6.046752695s" totalImagesPullingTime="3.463375701s" totalInitContainerRuntime="1.090289073s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:57:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:57:19.043947262 +0000 UTC m=+346.923336694" watchObservedRunningTime="2026-09-10 11:57:19.046752695 +0000 UTC m=+346.926142126" Sep 10 11:57:30.019193 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:30.019094 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-58rlx" Sep 10 11:57:43.091227 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.091183 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j"] Sep 10 11:57:43.096326 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.096305 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" Sep 10 11:57:43.100858 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.100833 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 11:57:43.101172 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.101135 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 11:57:43.102121 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.102087 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 10 11:57:43.102121 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.102101 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 11:57:43.102321 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.102101 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gl5hz\"" Sep 10 11:57:43.105071 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.105044 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j"] Sep 10 11:57:43.221258 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.221221 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9d78j"] Sep 10 11:57:43.230550 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.228483 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" Sep 10 11:57:43.230550 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.229774 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae94e46e-b2f2-4d01-971c-81b2866a4686-tmp\") pod \"kube-auth-proxy-5986dd8d8b-tx97j\" (UID: \"ae94e46e-b2f2-4d01-971c-81b2866a4686\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" Sep 10 11:57:43.230550 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.229832 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae94e46e-b2f2-4d01-971c-81b2866a4686-tls-certs\") pod \"kube-auth-proxy-5986dd8d8b-tx97j\" (UID: \"ae94e46e-b2f2-4d01-971c-81b2866a4686\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" Sep 10 11:57:43.230550 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.229895 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2krl\" (UniqueName: \"kubernetes.io/projected/ae94e46e-b2f2-4d01-971c-81b2866a4686-kube-api-access-l2krl\") pod \"kube-auth-proxy-5986dd8d8b-tx97j\" (UID: \"ae94e46e-b2f2-4d01-971c-81b2866a4686\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" Sep 10 11:57:43.233688 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.233429 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 11:57:43.233971 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.233947 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8svg4\"" Sep 10 11:57:43.248014 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.247977 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9d78j"] Sep 10 11:57:43.330473 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.330444 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3650596d-f3bb-4d38-8c6c-07c6fd3a5407-cert\") pod \"odh-model-controller-7998c67dc4-9d78j\" (UID: \"3650596d-f3bb-4d38-8c6c-07c6fd3a5407\") " pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" Sep 10 11:57:43.330657 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.330486 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9ns2\" (UniqueName: \"kubernetes.io/projected/3650596d-f3bb-4d38-8c6c-07c6fd3a5407-kube-api-access-g9ns2\") pod \"odh-model-controller-7998c67dc4-9d78j\" (UID: \"3650596d-f3bb-4d38-8c6c-07c6fd3a5407\") " pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" Sep 10 11:57:43.330657 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.330525 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae94e46e-b2f2-4d01-971c-81b2866a4686-tmp\") pod \"kube-auth-proxy-5986dd8d8b-tx97j\" (UID: \"ae94e46e-b2f2-4d01-971c-81b2866a4686\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" Sep 10 11:57:43.330657 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.330587 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae94e46e-b2f2-4d01-971c-81b2866a4686-tls-certs\") pod \"kube-auth-proxy-5986dd8d8b-tx97j\" (UID: \"ae94e46e-b2f2-4d01-971c-81b2866a4686\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" Sep 10 11:57:43.330657 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.330642 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2krl\" (UniqueName: \"kubernetes.io/projected/ae94e46e-b2f2-4d01-971c-81b2866a4686-kube-api-access-l2krl\") pod \"kube-auth-proxy-5986dd8d8b-tx97j\" (UID: \"ae94e46e-b2f2-4d01-971c-81b2866a4686\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" Sep 10 11:57:43.333008 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.332982 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae94e46e-b2f2-4d01-971c-81b2866a4686-tmp\") pod \"kube-auth-proxy-5986dd8d8b-tx97j\" (UID: \"ae94e46e-b2f2-4d01-971c-81b2866a4686\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" Sep 10 11:57:43.333367 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.333345 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae94e46e-b2f2-4d01-971c-81b2866a4686-tls-certs\") pod \"kube-auth-proxy-5986dd8d8b-tx97j\" (UID: \"ae94e46e-b2f2-4d01-971c-81b2866a4686\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" Sep 10 11:57:43.344467 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.344360 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2krl\" (UniqueName: \"kubernetes.io/projected/ae94e46e-b2f2-4d01-971c-81b2866a4686-kube-api-access-l2krl\") pod \"kube-auth-proxy-5986dd8d8b-tx97j\" (UID: \"ae94e46e-b2f2-4d01-971c-81b2866a4686\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" Sep 10 11:57:43.406319 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.406279 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" Sep 10 11:57:43.432074 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.432039 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3650596d-f3bb-4d38-8c6c-07c6fd3a5407-cert\") pod \"odh-model-controller-7998c67dc4-9d78j\" (UID: \"3650596d-f3bb-4d38-8c6c-07c6fd3a5407\") " pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" Sep 10 11:57:43.432245 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.432086 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9ns2\" (UniqueName: \"kubernetes.io/projected/3650596d-f3bb-4d38-8c6c-07c6fd3a5407-kube-api-access-g9ns2\") pod \"odh-model-controller-7998c67dc4-9d78j\" (UID: \"3650596d-f3bb-4d38-8c6c-07c6fd3a5407\") " pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" Sep 10 11:57:43.432457 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:57:43.432424 2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 10 11:57:43.432572 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:57:43.432508 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3650596d-f3bb-4d38-8c6c-07c6fd3a5407-cert podName:3650596d-f3bb-4d38-8c6c-07c6fd3a5407 nodeName:}" failed. No retries permitted until 2026-09-10 11:57:43.932486171 +0000 UTC m=+371.811875583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3650596d-f3bb-4d38-8c6c-07c6fd3a5407-cert") pod "odh-model-controller-7998c67dc4-9d78j" (UID: "3650596d-f3bb-4d38-8c6c-07c6fd3a5407") : secret "odh-model-controller-webhook-cert" not found Sep 10 11:57:43.442457 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.442422 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9ns2\" (UniqueName: \"kubernetes.io/projected/3650596d-f3bb-4d38-8c6c-07c6fd3a5407-kube-api-access-g9ns2\") pod \"odh-model-controller-7998c67dc4-9d78j\" (UID: \"3650596d-f3bb-4d38-8c6c-07c6fd3a5407\") " pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" Sep 10 11:57:43.683518 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.683443 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cfkvr"] Sep 10 11:57:43.687792 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.687770 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-cfkvr" Sep 10 11:57:43.691296 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.691267 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 10 11:57:43.697501 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.697481 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j"] Sep 10 11:57:43.699610 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:57:43.699588 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae94e46e_b2f2_4d01_971c_81b2866a4686.slice/crio-5539861ac7bac2e6bffd21d3aa42de2f19ea066c52b46ace2324a89d34c25137 WatchSource:0}: Error finding container 5539861ac7bac2e6bffd21d3aa42de2f19ea066c52b46ace2324a89d34c25137: Status 404 returned error can't find the container with id 5539861ac7bac2e6bffd21d3aa42de2f19ea066c52b46ace2324a89d34c25137 Sep 10 11:57:43.703261 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.703240 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cfkvr"] Sep 10 11:57:43.835548 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.835514 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4qgd\" (UniqueName: \"kubernetes.io/projected/37b9575a-5b55-4dfd-b70b-a254bedee0c8-kube-api-access-z4qgd\") pod \"model-serving-api-54d89bd5fc-cfkvr\" (UID: \"37b9575a-5b55-4dfd-b70b-a254bedee0c8\") " pod="opendatahub/model-serving-api-54d89bd5fc-cfkvr" Sep 10 11:57:43.835750 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.835652 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b9575a-5b55-4dfd-b70b-a254bedee0c8-tls-certs\") pod \"model-serving-api-54d89bd5fc-cfkvr\" (UID: \"37b9575a-5b55-4dfd-b70b-a254bedee0c8\") " pod="opendatahub/model-serving-api-54d89bd5fc-cfkvr" Sep 10 11:57:43.937115 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.937085 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b9575a-5b55-4dfd-b70b-a254bedee0c8-tls-certs\") pod \"model-serving-api-54d89bd5fc-cfkvr\" (UID: \"37b9575a-5b55-4dfd-b70b-a254bedee0c8\") " pod="opendatahub/model-serving-api-54d89bd5fc-cfkvr" Sep 10 11:57:43.937307 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.937123 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3650596d-f3bb-4d38-8c6c-07c6fd3a5407-cert\") pod \"odh-model-controller-7998c67dc4-9d78j\" (UID: \"3650596d-f3bb-4d38-8c6c-07c6fd3a5407\") " pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" Sep 10 11:57:43.937307 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.937154 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4qgd\" (UniqueName: \"kubernetes.io/projected/37b9575a-5b55-4dfd-b70b-a254bedee0c8-kube-api-access-z4qgd\") pod \"model-serving-api-54d89bd5fc-cfkvr\" (UID: \"37b9575a-5b55-4dfd-b70b-a254bedee0c8\") " pod="opendatahub/model-serving-api-54d89bd5fc-cfkvr" Sep 10 11:57:43.939869 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.939844 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3650596d-f3bb-4d38-8c6c-07c6fd3a5407-cert\") pod \"odh-model-controller-7998c67dc4-9d78j\" (UID: \"3650596d-f3bb-4d38-8c6c-07c6fd3a5407\") " pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" Sep 10 11:57:43.939989 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.939916 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37b9575a-5b55-4dfd-b70b-a254bedee0c8-tls-certs\") pod \"model-serving-api-54d89bd5fc-cfkvr\" (UID: \"37b9575a-5b55-4dfd-b70b-a254bedee0c8\") " pod="opendatahub/model-serving-api-54d89bd5fc-cfkvr" Sep 10 11:57:43.956608 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:43.956586 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4qgd\" (UniqueName: \"kubernetes.io/projected/37b9575a-5b55-4dfd-b70b-a254bedee0c8-kube-api-access-z4qgd\") pod \"model-serving-api-54d89bd5fc-cfkvr\" (UID: \"37b9575a-5b55-4dfd-b70b-a254bedee0c8\") " pod="opendatahub/model-serving-api-54d89bd5fc-cfkvr" Sep 10 11:57:44.000336 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.000308 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-cfkvr" Sep 10 11:57:44.118504 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.118468 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" event={"ID":"ae94e46e-b2f2-4d01-971c-81b2866a4686","Type":"ContainerStarted","Data":"5539861ac7bac2e6bffd21d3aa42de2f19ea066c52b46ace2324a89d34c25137"} Sep 10 11:57:44.142833 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.142797 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" Sep 10 11:57:44.301263 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.301238 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cfkvr"] Sep 10 11:57:44.302086 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:57:44.302058 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b9575a_5b55_4dfd_b70b_a254bedee0c8.slice/crio-eb8186365ec8f4d805566ed18301ae7fa13cebd229fa24ad961dc7e10bdc32e5 WatchSource:0}: Error finding container eb8186365ec8f4d805566ed18301ae7fa13cebd229fa24ad961dc7e10bdc32e5: Status 404 returned error can't find the container with id eb8186365ec8f4d805566ed18301ae7fa13cebd229fa24ad961dc7e10bdc32e5 Sep 10 11:57:44.427202 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.427164 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9d78j"] Sep 10 11:57:44.430884 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:57:44.430855 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3650596d_f3bb_4d38_8c6c_07c6fd3a5407.slice/crio-56953b0ded9e3276213b7dd4cf249dae8ae7bb3a69e1cb4a1c76a88e60e7f4fd WatchSource:0}: Error finding container 56953b0ded9e3276213b7dd4cf249dae8ae7bb3a69e1cb4a1c76a88e60e7f4fd: Status 404 returned error can't find the container with id 56953b0ded9e3276213b7dd4cf249dae8ae7bb3a69e1cb4a1c76a88e60e7f4fd Sep 10 11:57:44.748048 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.748012 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2"] Sep 10 11:57:44.751629 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.751607 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.754827 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.754804 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 11:57:44.754966 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.754804 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 11:57:44.755035 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.754975 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 11:57:44.755163 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.755105 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-r86vv\"" Sep 10 11:57:44.755380 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.755364 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 11:57:44.770283 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.770257 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2"] Sep 10 11:57:44.845175 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.845102 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.845334 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.845174 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.845334 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.845229 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.845334 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.845313 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.845536 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.845348 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.845536 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.845383 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jlg\" (UniqueName: \"kubernetes.io/projected/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-kube-api-access-97jlg\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.845536 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.845417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.946793 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.946395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.946793 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.946455 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.946793 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.946499 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.946793 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.946531 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.946793 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.946551 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.946793 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.946575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jlg\" (UniqueName: \"kubernetes.io/projected/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-kube-api-access-97jlg\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.946793 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.946601 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.948418 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.948360 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.950637 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.950546 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.955501 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.955247 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.956479 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.956398 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.958573 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.958541 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.959344 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.959306 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jlg\" (UniqueName: \"kubernetes.io/projected/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-kube-api-access-97jlg\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:44.961027 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:44.960986 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lswb2\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:45.064221 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:45.064098 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:45.128319 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:45.128279 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-cfkvr" event={"ID":"37b9575a-5b55-4dfd-b70b-a254bedee0c8","Type":"ContainerStarted","Data":"eb8186365ec8f4d805566ed18301ae7fa13cebd229fa24ad961dc7e10bdc32e5"} Sep 10 11:57:45.131248 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:45.131215 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" event={"ID":"3650596d-f3bb-4d38-8c6c-07c6fd3a5407","Type":"ContainerStarted","Data":"56953b0ded9e3276213b7dd4cf249dae8ae7bb3a69e1cb4a1c76a88e60e7f4fd"} Sep 10 11:57:45.384360 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:45.383048 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2"] Sep 10 11:57:45.385759 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:57:45.385725 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce72aa3f_a8d4_4007_bacf_f9177b7c2360.slice/crio-50f6e288e3f8c5892c309525208336d1592b3bf2c036bd6f57fa7a5883b603a0 WatchSource:0}: Error finding container 50f6e288e3f8c5892c309525208336d1592b3bf2c036bd6f57fa7a5883b603a0: Status 404 returned error can't find the container with id 50f6e288e3f8c5892c309525208336d1592b3bf2c036bd6f57fa7a5883b603a0 Sep 10 11:57:46.139924 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:46.139883 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" event={"ID":"ce72aa3f-a8d4-4007-bacf-f9177b7c2360","Type":"ContainerStarted","Data":"50f6e288e3f8c5892c309525208336d1592b3bf2c036bd6f57fa7a5883b603a0"} Sep 10 11:57:48.155080 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:48.154107 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-cfkvr" event={"ID":"37b9575a-5b55-4dfd-b70b-a254bedee0c8","Type":"ContainerStarted","Data":"eabd45272224a1792db2336b54c90f1fef7592945ebfd8528b656837f1897f52"} Sep 10 11:57:48.155080 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:48.154965 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-cfkvr" Sep 10 11:57:48.158164 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:48.157621 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" event={"ID":"ae94e46e-b2f2-4d01-971c-81b2866a4686","Type":"ContainerStarted","Data":"bdc67033e00f7482c2e4c82748bcc51d09869f789b2deb6d67e3095abdcad06c"} Sep 10 11:57:48.200644 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:48.199638 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-cfkvr" podStartSLOduration=1.658243935 podStartE2EDuration="5.19961923s" totalImagesPullingTime="3.541375295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:57:48.177964663 +0000 UTC m=+376.057354097" watchObservedRunningTime="2026-09-10 11:57:48.19961923 +0000 UTC m=+376.079008663" Sep 10 11:57:48.201774 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:48.201576 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-tx97j" podStartSLOduration=1.057742036 podStartE2EDuration="5.201562466s" totalImagesPullingTime="4.14382043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:57:48.197116634 +0000 UTC m=+376.076506070" watchObservedRunningTime="2026-09-10 11:57:48.201562466 +0000 UTC m=+376.080951902" Sep 10 11:57:50.999901 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:50.999851 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 11:57:51.000262 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:50.999933 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 11:57:51.174979 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:51.174937 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" event={"ID":"3650596d-f3bb-4d38-8c6c-07c6fd3a5407","Type":"ContainerStarted","Data":"20776a667f2a0e5b3b15a8bfc4ae007819f882e54af9633df895487a13a987a2"} Sep 10 11:57:51.175180 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:51.175033 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" Sep 10 11:57:51.176686 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:51.176662 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" event={"ID":"ce72aa3f-a8d4-4007-bacf-f9177b7c2360","Type":"ContainerStarted","Data":"e09a7c655ceb24211e1dc8ec9bd41434919a7700bf2e3f65d430122c543450ee"} Sep 10 11:57:51.176840 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:51.176820 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:51.197523 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:51.197481 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" podStartSLOduration=1.629901395 podStartE2EDuration="8.197466733s" totalImagesPullingTime="6.567565338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:57:51.195813856 +0000 UTC m=+379.075203288" watchObservedRunningTime="2026-09-10 11:57:51.197466733 +0000 UTC m=+379.076856164" Sep 10 11:57:51.222766 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:51.222723 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" podStartSLOduration=1.611629449 podStartE2EDuration="7.222708218s" totalImagesPullingTime="5.611078769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:57:51.22036739 +0000 UTC m=+379.099756821" watchObservedRunningTime="2026-09-10 11:57:51.222708218 +0000 UTC m=+379.102097650" Sep 10 11:57:52.187204 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:52.187169 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:55.512491 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.512460 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-66rzq"] Sep 10 11:57:55.515566 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.515550 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" Sep 10 11:57:55.519455 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.519425 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 10 11:57:55.519744 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.519724 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dv95h\"" Sep 10 11:57:55.532685 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.532649 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-66rzq"] Sep 10 11:57:55.538434 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.538410 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j"] Sep 10 11:57:55.541555 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.541536 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j" Sep 10 11:57:55.544900 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.544880 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 10 11:57:55.547810 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.547781 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-84s5c\"" Sep 10 11:57:55.560412 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.560380 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j"] Sep 10 11:57:55.647022 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.646994 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01ccad84-896c-4704-85ab-3dee8a5caf02-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-px45j\" (UID: \"01ccad84-896c-4704-85ab-3dee8a5caf02\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j" Sep 10 11:57:55.647022 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.647034 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/446e89f1-88a5-4676-a504-fd413ded2aa1-cert\") pod \"kserve-controller-manager-d6f4b65b8-66rzq\" (UID: \"446e89f1-88a5-4676-a504-fd413ded2aa1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" Sep 10 11:57:55.647228 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.647054 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgj27\" (UniqueName: \"kubernetes.io/projected/01ccad84-896c-4704-85ab-3dee8a5caf02-kube-api-access-kgj27\") pod \"llmisvc-controller-manager-777c6bd7d5-px45j\" (UID: \"01ccad84-896c-4704-85ab-3dee8a5caf02\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j" Sep 10 11:57:55.647228 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.647175 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tsdp\" (UniqueName: \"kubernetes.io/projected/446e89f1-88a5-4676-a504-fd413ded2aa1-kube-api-access-2tsdp\") pod \"kserve-controller-manager-d6f4b65b8-66rzq\" (UID: \"446e89f1-88a5-4676-a504-fd413ded2aa1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" Sep 10 11:57:55.748525 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.748491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01ccad84-896c-4704-85ab-3dee8a5caf02-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-px45j\" (UID: \"01ccad84-896c-4704-85ab-3dee8a5caf02\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j" Sep 10 11:57:55.748682 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.748535 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/446e89f1-88a5-4676-a504-fd413ded2aa1-cert\") pod \"kserve-controller-manager-d6f4b65b8-66rzq\" (UID: \"446e89f1-88a5-4676-a504-fd413ded2aa1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" Sep 10 11:57:55.748682 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.748556 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgj27\" (UniqueName: \"kubernetes.io/projected/01ccad84-896c-4704-85ab-3dee8a5caf02-kube-api-access-kgj27\") pod \"llmisvc-controller-manager-777c6bd7d5-px45j\" (UID: \"01ccad84-896c-4704-85ab-3dee8a5caf02\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j" Sep 10 11:57:55.748682 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.748599 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tsdp\" (UniqueName: \"kubernetes.io/projected/446e89f1-88a5-4676-a504-fd413ded2aa1-kube-api-access-2tsdp\") pod \"kserve-controller-manager-d6f4b65b8-66rzq\" (UID: \"446e89f1-88a5-4676-a504-fd413ded2aa1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" Sep 10 11:57:55.748682 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:57:55.748654 2582 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 10 11:57:55.748822 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:57:55.748731 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/446e89f1-88a5-4676-a504-fd413ded2aa1-cert podName:446e89f1-88a5-4676-a504-fd413ded2aa1 nodeName:}" failed. No retries permitted until 2026-09-10 11:57:56.248711804 +0000 UTC m=+384.128101214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/446e89f1-88a5-4676-a504-fd413ded2aa1-cert") pod "kserve-controller-manager-d6f4b65b8-66rzq" (UID: "446e89f1-88a5-4676-a504-fd413ded2aa1") : secret "kserve-webhook-server-cert" not found Sep 10 11:57:55.751101 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.751083 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01ccad84-896c-4704-85ab-3dee8a5caf02-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-px45j\" (UID: \"01ccad84-896c-4704-85ab-3dee8a5caf02\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j" Sep 10 11:57:55.762466 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.762445 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tsdp\" (UniqueName: \"kubernetes.io/projected/446e89f1-88a5-4676-a504-fd413ded2aa1-kube-api-access-2tsdp\") pod \"kserve-controller-manager-d6f4b65b8-66rzq\" (UID: \"446e89f1-88a5-4676-a504-fd413ded2aa1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" Sep 10 11:57:55.765801 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.765745 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgj27\" (UniqueName: \"kubernetes.io/projected/01ccad84-896c-4704-85ab-3dee8a5caf02-kube-api-access-kgj27\") pod \"llmisvc-controller-manager-777c6bd7d5-px45j\" (UID: \"01ccad84-896c-4704-85ab-3dee8a5caf02\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j" Sep 10 11:57:55.852451 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:55.852418 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j" Sep 10 11:57:56.133705 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:56.133681 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j"] Sep 10 11:57:56.136314 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:57:56.136274 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod01ccad84_896c_4704_85ab_3dee8a5caf02.slice/crio-f2fa5efeefe9aa506d7f4c785d4dd1498bc854eb4d01b1bcfbfc98189b10ab90 WatchSource:0}: Error finding container f2fa5efeefe9aa506d7f4c785d4dd1498bc854eb4d01b1bcfbfc98189b10ab90: Status 404 returned error can't find the container with id f2fa5efeefe9aa506d7f4c785d4dd1498bc854eb4d01b1bcfbfc98189b10ab90 Sep 10 11:57:56.200638 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:56.200608 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j" event={"ID":"01ccad84-896c-4704-85ab-3dee8a5caf02","Type":"ContainerStarted","Data":"f2fa5efeefe9aa506d7f4c785d4dd1498bc854eb4d01b1bcfbfc98189b10ab90"} Sep 10 11:57:56.253059 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:56.253033 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/446e89f1-88a5-4676-a504-fd413ded2aa1-cert\") pod \"kserve-controller-manager-d6f4b65b8-66rzq\" (UID: \"446e89f1-88a5-4676-a504-fd413ded2aa1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" Sep 10 11:57:56.255517 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:56.255500 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/446e89f1-88a5-4676-a504-fd413ded2aa1-cert\") pod \"kserve-controller-manager-d6f4b65b8-66rzq\" (UID: \"446e89f1-88a5-4676-a504-fd413ded2aa1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" Sep 10 11:57:56.428061 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:56.427980 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" Sep 10 11:57:56.720133 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:56.720112 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-66rzq"] Sep 10 11:57:56.721746 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:57:56.721716 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446e89f1_88a5_4676_a504_fd413ded2aa1.slice/crio-1a4d9bf9c63aa637e31c6afec18301bc98829af2519f223a02c4149e8d41fbcf WatchSource:0}: Error finding container 1a4d9bf9c63aa637e31c6afec18301bc98829af2519f223a02c4149e8d41fbcf: Status 404 returned error can't find the container with id 1a4d9bf9c63aa637e31c6afec18301bc98829af2519f223a02c4149e8d41fbcf Sep 10 11:57:57.206571 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.206537 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" event={"ID":"446e89f1-88a5-4676-a504-fd413ded2aa1","Type":"ContainerStarted","Data":"1a4d9bf9c63aa637e31c6afec18301bc98829af2519f223a02c4149e8d41fbcf"} Sep 10 11:57:57.619032 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.618935 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2"] Sep 10 11:57:57.619239 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.619175 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" podUID="ce72aa3f-a8d4-4007-bacf-f9177b7c2360" containerName="discovery" containerID="cri-o://e09a7c655ceb24211e1dc8ec9bd41434919a7700bf2e3f65d430122c543450ee" gracePeriod=30 Sep 10 11:57:57.943052 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.942994 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:57.974200 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.974170 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-csr-dns-cert\") pod \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " Sep 10 11:57:57.974365 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.974221 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-token\") pod \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " Sep 10 11:57:57.974365 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.974252 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97jlg\" (UniqueName: \"kubernetes.io/projected/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-kube-api-access-97jlg\") pod \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " Sep 10 11:57:57.974365 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.974307 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-csr-ca-configmap\") pod \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " Sep 10 11:57:57.974590 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.974561 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-cacerts\") pod \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " Sep 10 11:57:57.974644 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.974609 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-local-certs\") pod \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " Sep 10 11:57:57.974696 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.974644 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-kubeconfig\") pod \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\" (UID: \"ce72aa3f-a8d4-4007-bacf-f9177b7c2360\") " Sep 10 11:57:57.974743 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.974719 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ce72aa3f-a8d4-4007-bacf-f9177b7c2360" (UID: "ce72aa3f-a8d4-4007-bacf-f9177b7c2360"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 11:57:57.974963 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.974943 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-csr-ca-configmap\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:57:57.978889 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.978029 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-kube-api-access-97jlg" (OuterVolumeSpecName: "kube-api-access-97jlg") pod "ce72aa3f-a8d4-4007-bacf-f9177b7c2360" (UID: "ce72aa3f-a8d4-4007-bacf-f9177b7c2360"). InnerVolumeSpecName "kube-api-access-97jlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 11:57:57.978889 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.978059 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-token" (OuterVolumeSpecName: "istio-token") pod "ce72aa3f-a8d4-4007-bacf-f9177b7c2360" (UID: "ce72aa3f-a8d4-4007-bacf-f9177b7c2360"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 11:57:57.979749 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.979250 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ce72aa3f-a8d4-4007-bacf-f9177b7c2360" (UID: "ce72aa3f-a8d4-4007-bacf-f9177b7c2360"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 11:57:57.979749 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.979510 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-cacerts" (OuterVolumeSpecName: "cacerts") pod "ce72aa3f-a8d4-4007-bacf-f9177b7c2360" (UID: "ce72aa3f-a8d4-4007-bacf-f9177b7c2360"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 11:57:57.979749 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.979674 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-local-certs" (OuterVolumeSpecName: "local-certs") pod "ce72aa3f-a8d4-4007-bacf-f9177b7c2360" (UID: "ce72aa3f-a8d4-4007-bacf-f9177b7c2360"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 11:57:57.980090 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:57.980065 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ce72aa3f-a8d4-4007-bacf-f9177b7c2360" (UID: "ce72aa3f-a8d4-4007-bacf-f9177b7c2360"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 11:57:58.076008 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.075969 2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-cacerts\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:57:58.076008 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.076005 2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-local-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:57:58.076250 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.076020 2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-kubeconfig\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:57:58.076250 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.076030 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-csr-dns-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:57:58.076250 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.076042 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:57:58.076250 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.076056 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97jlg\" (UniqueName: \"kubernetes.io/projected/ce72aa3f-a8d4-4007-bacf-f9177b7c2360-kube-api-access-97jlg\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:57:58.214069 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.213766 2582 generic.go:356] "Generic (PLEG): container finished" podID="ce72aa3f-a8d4-4007-bacf-f9177b7c2360" containerID="e09a7c655ceb24211e1dc8ec9bd41434919a7700bf2e3f65d430122c543450ee" exitCode=0 Sep 10 11:57:58.214069 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.213821 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" event={"ID":"ce72aa3f-a8d4-4007-bacf-f9177b7c2360","Type":"ContainerDied","Data":"e09a7c655ceb24211e1dc8ec9bd41434919a7700bf2e3f65d430122c543450ee"} Sep 10 11:57:58.214069 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.213904 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" event={"ID":"ce72aa3f-a8d4-4007-bacf-f9177b7c2360","Type":"ContainerDied","Data":"50f6e288e3f8c5892c309525208336d1592b3bf2c036bd6f57fa7a5883b603a0"} Sep 10 11:57:58.214069 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.213935 2582 scope.go:117] "RemoveContainer" containerID="e09a7c655ceb24211e1dc8ec9bd41434919a7700bf2e3f65d430122c543450ee" Sep 10 11:57:58.214069 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.213980 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2" Sep 10 11:57:58.247838 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.247807 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2"] Sep 10 11:57:58.259588 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.259559 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lswb2"] Sep 10 11:57:58.262697 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.262675 2582 scope.go:117] "RemoveContainer" containerID="e09a7c655ceb24211e1dc8ec9bd41434919a7700bf2e3f65d430122c543450ee" Sep 10 11:57:58.263066 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:57:58.263043 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09a7c655ceb24211e1dc8ec9bd41434919a7700bf2e3f65d430122c543450ee\": container with ID starting with e09a7c655ceb24211e1dc8ec9bd41434919a7700bf2e3f65d430122c543450ee not found: ID does not exist" containerID="e09a7c655ceb24211e1dc8ec9bd41434919a7700bf2e3f65d430122c543450ee" Sep 10 11:57:58.263181 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.263076 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09a7c655ceb24211e1dc8ec9bd41434919a7700bf2e3f65d430122c543450ee"} err="failed to get container status \"e09a7c655ceb24211e1dc8ec9bd41434919a7700bf2e3f65d430122c543450ee\": rpc error: code = NotFound desc = could not find container \"e09a7c655ceb24211e1dc8ec9bd41434919a7700bf2e3f65d430122c543450ee\": container with ID starting with e09a7c655ceb24211e1dc8ec9bd41434919a7700bf2e3f65d430122c543450ee not found: ID does not exist" Sep 10 11:57:58.630740 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:57:58.630651 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce72aa3f-a8d4-4007-bacf-f9177b7c2360" path="/var/lib/kubelet/pods/ce72aa3f-a8d4-4007-bacf-f9177b7c2360/volumes" Sep 10 11:58:00.172632 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:00.172552 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-cfkvr" Sep 10 11:58:00.227267 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:00.227232 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j" event={"ID":"01ccad84-896c-4704-85ab-3dee8a5caf02","Type":"ContainerStarted","Data":"d531de14af7968d8a76715cb15ad7d8ae60a3b8a8be610c7ba74b55fa1585f73"} Sep 10 11:58:00.227421 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:00.227359 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j" Sep 10 11:58:00.228878 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:00.228848 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" event={"ID":"446e89f1-88a5-4676-a504-fd413ded2aa1","Type":"ContainerStarted","Data":"6c02092a5684cc29236e8753be43d6a40b9e128243fc0f03d32261b84edaca11"} Sep 10 11:58:00.228998 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:00.228985 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" Sep 10 11:58:00.255098 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:00.255053 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j" podStartSLOduration=1.565792713 podStartE2EDuration="5.255040737s" totalImagesPullingTime="3.689248024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:57:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:58:00.252605999 +0000 UTC m=+388.131995431" watchObservedRunningTime="2026-09-10 11:58:00.255040737 +0000 UTC m=+388.134430169" Sep 10 11:58:00.281895 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:00.281844 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" podStartSLOduration=2.812587678 podStartE2EDuration="5.281830857s" totalImagesPullingTime="2.469243179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:57:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:58:00.280714221 +0000 UTC m=+388.160103653" watchObservedRunningTime="2026-09-10 11:58:00.281830857 +0000 UTC m=+388.161220288" Sep 10 11:58:02.188894 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:02.188869 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9d78j" Sep 10 11:58:21.717302 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.717265 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78df4fdf45-kp5dt"] Sep 10 11:58:21.717705 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.717576 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce72aa3f-a8d4-4007-bacf-f9177b7c2360" containerName="discovery" Sep 10 11:58:21.717705 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.717587 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce72aa3f-a8d4-4007-bacf-f9177b7c2360" containerName="discovery" Sep 10 11:58:21.717705 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.717667 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce72aa3f-a8d4-4007-bacf-f9177b7c2360" containerName="discovery" Sep 10 11:58:21.788432 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.788398 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78df4fdf45-kp5dt"] Sep 10 11:58:21.788570 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.788487 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.872516 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.872486 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkmft\" (UniqueName: \"kubernetes.io/projected/440d2d33-8ff9-422b-8e0c-dbb95296c145-kube-api-access-jkmft\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.872650 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.872527 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/440d2d33-8ff9-422b-8e0c-dbb95296c145-console-serving-cert\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.872650 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.872552 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/440d2d33-8ff9-422b-8e0c-dbb95296c145-service-ca\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.872650 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.872634 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440d2d33-8ff9-422b-8e0c-dbb95296c145-trusted-ca-bundle\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.872751 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.872676 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/440d2d33-8ff9-422b-8e0c-dbb95296c145-console-config\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.872751 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.872701 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/440d2d33-8ff9-422b-8e0c-dbb95296c145-oauth-serving-cert\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.872751 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.872739 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/440d2d33-8ff9-422b-8e0c-dbb95296c145-console-oauth-config\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.973476 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.973399 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkmft\" (UniqueName: \"kubernetes.io/projected/440d2d33-8ff9-422b-8e0c-dbb95296c145-kube-api-access-jkmft\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.973476 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.973431 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/440d2d33-8ff9-422b-8e0c-dbb95296c145-console-serving-cert\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.973476 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.973448 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/440d2d33-8ff9-422b-8e0c-dbb95296c145-service-ca\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.973476 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.973476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440d2d33-8ff9-422b-8e0c-dbb95296c145-trusted-ca-bundle\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.973790 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.973497 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/440d2d33-8ff9-422b-8e0c-dbb95296c145-console-config\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.973790 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.973514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/440d2d33-8ff9-422b-8e0c-dbb95296c145-oauth-serving-cert\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.973790 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.973538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/440d2d33-8ff9-422b-8e0c-dbb95296c145-console-oauth-config\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.974314 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.974290 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/440d2d33-8ff9-422b-8e0c-dbb95296c145-console-config\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.974429 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.974408 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/440d2d33-8ff9-422b-8e0c-dbb95296c145-service-ca\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.974470 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.974416 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/440d2d33-8ff9-422b-8e0c-dbb95296c145-oauth-serving-cert\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.974504 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.974481 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440d2d33-8ff9-422b-8e0c-dbb95296c145-trusted-ca-bundle\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.976548 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.976521 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/440d2d33-8ff9-422b-8e0c-dbb95296c145-console-serving-cert\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.976636 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.976575 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/440d2d33-8ff9-422b-8e0c-dbb95296c145-console-oauth-config\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:21.982919 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:21.982900 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkmft\" (UniqueName: \"kubernetes.io/projected/440d2d33-8ff9-422b-8e0c-dbb95296c145-kube-api-access-jkmft\") pod \"console-78df4fdf45-kp5dt\" (UID: \"440d2d33-8ff9-422b-8e0c-dbb95296c145\") " pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:22.097021 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:22.096982 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:22.389919 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:22.389841 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78df4fdf45-kp5dt"] Sep 10 11:58:22.393650 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:58:22.393605 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440d2d33_8ff9_422b_8e0c_dbb95296c145.slice/crio-0203d7bc973881cbb1f5b69677d0682f36d4a47e05b72165e10ce9a4fea70452 WatchSource:0}: Error finding container 0203d7bc973881cbb1f5b69677d0682f36d4a47e05b72165e10ce9a4fea70452: Status 404 returned error can't find the container with id 0203d7bc973881cbb1f5b69677d0682f36d4a47e05b72165e10ce9a4fea70452 Sep 10 11:58:23.336414 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:23.336380 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78df4fdf45-kp5dt" event={"ID":"440d2d33-8ff9-422b-8e0c-dbb95296c145","Type":"ContainerStarted","Data":"ce85b814f1149672f7f5820008fd5debc16a08a90a4d4f7eb690858bbfc4bb86"} Sep 10 11:58:23.336802 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:23.336456 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78df4fdf45-kp5dt" event={"ID":"440d2d33-8ff9-422b-8e0c-dbb95296c145","Type":"ContainerStarted","Data":"0203d7bc973881cbb1f5b69677d0682f36d4a47e05b72165e10ce9a4fea70452"} Sep 10 11:58:23.358201 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:23.358125 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78df4fdf45-kp5dt" podStartSLOduration=2.35811025 podStartE2EDuration="2.35811025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:58:23.356566933 +0000 UTC m=+411.235956364" watchObservedRunningTime="2026-09-10 11:58:23.35811025 +0000 UTC m=+411.237499682" Sep 10 11:58:31.234900 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:31.234870 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-px45j" Sep 10 11:58:31.238238 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:31.238219 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-66rzq" Sep 10 11:58:32.097963 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:32.097923 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:32.098194 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:32.097979 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:32.102818 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:32.102795 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:32.382276 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:32.382194 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78df4fdf45-kp5dt" Sep 10 11:58:32.436291 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:32.436262 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b6766bf8d-4c2rg"] Sep 10 11:58:57.462465 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.462384 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b6766bf8d-4c2rg" podUID="611baf38-a738-4227-b569-e1d6183d2b45" containerName="console" containerID="cri-o://fc3b850fa9cfca0bb8fa15a41fccaad0c9e997df624737f3b822afb1daa40bf9" gracePeriod=15 Sep 10 11:58:57.712565 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.712513 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:58:57.775695 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.775668 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/611baf38-a738-4227-b569-e1d6183d2b45-console-oauth-config\") pod \"611baf38-a738-4227-b569-e1d6183d2b45\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " Sep 10 11:58:57.775845 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.775709 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-oauth-serving-cert\") pod \"611baf38-a738-4227-b569-e1d6183d2b45\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " Sep 10 11:58:57.775913 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.775846 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-console-config\") pod \"611baf38-a738-4227-b569-e1d6183d2b45\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " Sep 10 11:58:57.775913 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.775892 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-service-ca\") pod \"611baf38-a738-4227-b569-e1d6183d2b45\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " Sep 10 11:58:57.776020 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.775933 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/611baf38-a738-4227-b569-e1d6183d2b45-console-serving-cert\") pod \"611baf38-a738-4227-b569-e1d6183d2b45\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " Sep 10 11:58:57.776020 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.775991 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjl2d\" (UniqueName: \"kubernetes.io/projected/611baf38-a738-4227-b569-e1d6183d2b45-kube-api-access-hjl2d\") pod \"611baf38-a738-4227-b569-e1d6183d2b45\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " Sep 10 11:58:57.776114 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.776026 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-trusted-ca-bundle\") pod \"611baf38-a738-4227-b569-e1d6183d2b45\" (UID: \"611baf38-a738-4227-b569-e1d6183d2b45\") " Sep 10 11:58:57.776273 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.776200 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "611baf38-a738-4227-b569-e1d6183d2b45" (UID: "611baf38-a738-4227-b569-e1d6183d2b45"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 11:58:57.776273 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.776248 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-console-config" (OuterVolumeSpecName: "console-config") pod "611baf38-a738-4227-b569-e1d6183d2b45" (UID: "611baf38-a738-4227-b569-e1d6183d2b45"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 11:58:57.776461 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.776316 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-oauth-serving-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:58:57.776461 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.776322 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-service-ca" (OuterVolumeSpecName: "service-ca") pod "611baf38-a738-4227-b569-e1d6183d2b45" (UID: "611baf38-a738-4227-b569-e1d6183d2b45"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 11:58:57.776461 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.776332 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-console-config\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:58:57.776561 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.776525 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "611baf38-a738-4227-b569-e1d6183d2b45" (UID: "611baf38-a738-4227-b569-e1d6183d2b45"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 11:58:57.777971 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.777950 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/611baf38-a738-4227-b569-e1d6183d2b45-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "611baf38-a738-4227-b569-e1d6183d2b45" (UID: "611baf38-a738-4227-b569-e1d6183d2b45"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 11:58:57.778229 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.778200 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/611baf38-a738-4227-b569-e1d6183d2b45-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "611baf38-a738-4227-b569-e1d6183d2b45" (UID: "611baf38-a738-4227-b569-e1d6183d2b45"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 11:58:57.778378 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.778352 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611baf38-a738-4227-b569-e1d6183d2b45-kube-api-access-hjl2d" (OuterVolumeSpecName: "kube-api-access-hjl2d") pod "611baf38-a738-4227-b569-e1d6183d2b45" (UID: "611baf38-a738-4227-b569-e1d6183d2b45"). InnerVolumeSpecName "kube-api-access-hjl2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 11:58:57.876971 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.876943 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:58:57.876971 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.876965 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/611baf38-a738-4227-b569-e1d6183d2b45-console-serving-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:58:57.876971 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.876975 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjl2d\" (UniqueName: \"kubernetes.io/projected/611baf38-a738-4227-b569-e1d6183d2b45-kube-api-access-hjl2d\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:58:57.877196 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.876984 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611baf38-a738-4227-b569-e1d6183d2b45-trusted-ca-bundle\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:58:57.877196 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:57.876993 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/611baf38-a738-4227-b569-e1d6183d2b45-console-oauth-config\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:58:58.495803 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:58.495770 2582 generic.go:356] "Generic (PLEG): container finished" podID="611baf38-a738-4227-b569-e1d6183d2b45" containerID="fc3b850fa9cfca0bb8fa15a41fccaad0c9e997df624737f3b822afb1daa40bf9" exitCode=0 Sep 10 11:58:58.496196 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:58.495818 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b6766bf8d-4c2rg" Sep 10 11:58:58.496196 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:58.495837 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6766bf8d-4c2rg" event={"ID":"611baf38-a738-4227-b569-e1d6183d2b45","Type":"ContainerDied","Data":"fc3b850fa9cfca0bb8fa15a41fccaad0c9e997df624737f3b822afb1daa40bf9"} Sep 10 11:58:58.496196 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:58.495901 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b6766bf8d-4c2rg" event={"ID":"611baf38-a738-4227-b569-e1d6183d2b45","Type":"ContainerDied","Data":"c9382f8f200770c927b095928d378e9295f51f0941e57217209219555effffd1"} Sep 10 11:58:58.496196 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:58.495921 2582 scope.go:117] "RemoveContainer" containerID="fc3b850fa9cfca0bb8fa15a41fccaad0c9e997df624737f3b822afb1daa40bf9" Sep 10 11:58:58.505296 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:58.505278 2582 scope.go:117] "RemoveContainer" containerID="fc3b850fa9cfca0bb8fa15a41fccaad0c9e997df624737f3b822afb1daa40bf9" Sep 10 11:58:58.505570 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:58:58.505551 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3b850fa9cfca0bb8fa15a41fccaad0c9e997df624737f3b822afb1daa40bf9\": container with ID starting with fc3b850fa9cfca0bb8fa15a41fccaad0c9e997df624737f3b822afb1daa40bf9 not found: ID does not exist" containerID="fc3b850fa9cfca0bb8fa15a41fccaad0c9e997df624737f3b822afb1daa40bf9" Sep 10 11:58:58.505649 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:58.505578 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3b850fa9cfca0bb8fa15a41fccaad0c9e997df624737f3b822afb1daa40bf9"} err="failed to get container status \"fc3b850fa9cfca0bb8fa15a41fccaad0c9e997df624737f3b822afb1daa40bf9\": rpc error: code = NotFound desc = could not find container \"fc3b850fa9cfca0bb8fa15a41fccaad0c9e997df624737f3b822afb1daa40bf9\": container with ID starting with fc3b850fa9cfca0bb8fa15a41fccaad0c9e997df624737f3b822afb1daa40bf9 not found: ID does not exist" Sep 10 11:58:58.518237 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:58.518214 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b6766bf8d-4c2rg"] Sep 10 11:58:58.523539 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:58.523508 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b6766bf8d-4c2rg"] Sep 10 11:58:58.629394 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:58:58.629361 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="611baf38-a738-4227-b569-e1d6183d2b45" path="/var/lib/kubelet/pods/611baf38-a738-4227-b569-e1d6183d2b45/volumes" Sep 10 11:59:00.201675 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.201645 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5sn2n"] Sep 10 11:59:00.202159 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.202117 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="611baf38-a738-4227-b569-e1d6183d2b45" containerName="console" Sep 10 11:59:00.202159 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.202135 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="611baf38-a738-4227-b569-e1d6183d2b45" containerName="console" Sep 10 11:59:00.202280 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.202271 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="611baf38-a738-4227-b569-e1d6183d2b45" containerName="console" Sep 10 11:59:00.206834 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.206815 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5sn2n" Sep 10 11:59:00.210844 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.210824 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 11:59:00.211036 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.211021 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 11:59:00.211084 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.211040 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-86n98\"" Sep 10 11:59:00.212016 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.212000 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 10 11:59:00.222431 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.222407 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5sn2n"] Sep 10 11:59:00.298315 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.298281 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl7j7\" (UniqueName: \"kubernetes.io/projected/a99e6748-3296-4f88-bf0d-44b7370d72bb-kube-api-access-wl7j7\") pod \"dns-operator-controller-manager-7b84cd7c9c-5sn2n\" (UID: \"a99e6748-3296-4f88-bf0d-44b7370d72bb\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5sn2n" Sep 10 11:59:00.399448 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.399421 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl7j7\" (UniqueName: \"kubernetes.io/projected/a99e6748-3296-4f88-bf0d-44b7370d72bb-kube-api-access-wl7j7\") pod \"dns-operator-controller-manager-7b84cd7c9c-5sn2n\" (UID: \"a99e6748-3296-4f88-bf0d-44b7370d72bb\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5sn2n" Sep 10 11:59:00.408673 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.408591 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl7j7\" (UniqueName: \"kubernetes.io/projected/a99e6748-3296-4f88-bf0d-44b7370d72bb-kube-api-access-wl7j7\") pod \"dns-operator-controller-manager-7b84cd7c9c-5sn2n\" (UID: \"a99e6748-3296-4f88-bf0d-44b7370d72bb\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5sn2n" Sep 10 11:59:00.517237 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.517163 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5sn2n" Sep 10 11:59:00.793323 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:00.793268 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5sn2n"] Sep 10 11:59:00.795730 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:59:00.795702 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99e6748_3296_4f88_bf0d_44b7370d72bb.slice/crio-5eb46f46dc83c4bea4d46f1eb234fec7fddd7c57b69952f976e67a20b4f4f8bb WatchSource:0}: Error finding container 5eb46f46dc83c4bea4d46f1eb234fec7fddd7c57b69952f976e67a20b4f4f8bb: Status 404 returned error can't find the container with id 5eb46f46dc83c4bea4d46f1eb234fec7fddd7c57b69952f976e67a20b4f4f8bb Sep 10 11:59:01.510296 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:01.510265 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5sn2n" event={"ID":"a99e6748-3296-4f88-bf0d-44b7370d72bb","Type":"ContainerStarted","Data":"5eb46f46dc83c4bea4d46f1eb234fec7fddd7c57b69952f976e67a20b4f4f8bb"} Sep 10 11:59:02.254036 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:02.253997 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5"] Sep 10 11:59:02.257549 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:02.257528 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:02.261043 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:02.261021 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dn86c\"" Sep 10 11:59:02.277437 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:02.277412 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5"] Sep 10 11:59:02.316103 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:02.316066 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc74q\" (UniqueName: \"kubernetes.io/projected/3c80af4f-10ac-4f5c-8988-a65ca97371b9-kube-api-access-tc74q\") pod \"limitador-operator-controller-manager-74495c69f8-9bpb5\" (UID: \"3c80af4f-10ac-4f5c-8988-a65ca97371b9\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:02.416613 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:02.416569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc74q\" (UniqueName: \"kubernetes.io/projected/3c80af4f-10ac-4f5c-8988-a65ca97371b9-kube-api-access-tc74q\") pod \"limitador-operator-controller-manager-74495c69f8-9bpb5\" (UID: \"3c80af4f-10ac-4f5c-8988-a65ca97371b9\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:02.439828 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:02.439790 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc74q\" (UniqueName: \"kubernetes.io/projected/3c80af4f-10ac-4f5c-8988-a65ca97371b9-kube-api-access-tc74q\") pod \"limitador-operator-controller-manager-74495c69f8-9bpb5\" (UID: \"3c80af4f-10ac-4f5c-8988-a65ca97371b9\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:02.570023 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:02.569881 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:03.521604 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:03.521575 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5sn2n" event={"ID":"a99e6748-3296-4f88-bf0d-44b7370d72bb","Type":"ContainerStarted","Data":"0a2392ad8214d03042688b194d3eac7d4daee3b7af4b3f344c94f9f2fc26cbf2"} Sep 10 11:59:03.521795 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:03.521703 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5sn2n" Sep 10 11:59:03.544891 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:03.544845 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5sn2n" podStartSLOduration=0.938429265 podStartE2EDuration="3.544829878s" totalImagesPullingTime="2.606400613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:59:03.544608966 +0000 UTC m=+451.423998398" watchObservedRunningTime="2026-09-10 11:59:03.544829878 +0000 UTC m=+451.424219311" Sep 10 11:59:03.608595 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:03.608569 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5"] Sep 10 11:59:03.610465 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:59:03.610438 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c80af4f_10ac_4f5c_8988_a65ca97371b9.slice/crio-ecfaacfae3b793463b40b60c94696e7948e3a336978cb8a2ac19f1a006c86e63 WatchSource:0}: Error finding container ecfaacfae3b793463b40b60c94696e7948e3a336978cb8a2ac19f1a006c86e63: Status 404 returned error can't find the container with id ecfaacfae3b793463b40b60c94696e7948e3a336978cb8a2ac19f1a006c86e63 Sep 10 11:59:04.528724 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:04.528686 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" event={"ID":"3c80af4f-10ac-4f5c-8988-a65ca97371b9","Type":"ContainerStarted","Data":"ecfaacfae3b793463b40b60c94696e7948e3a336978cb8a2ac19f1a006c86e63"} Sep 10 11:59:05.534519 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:05.534481 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" event={"ID":"3c80af4f-10ac-4f5c-8988-a65ca97371b9","Type":"ContainerStarted","Data":"c3ae0376e477cf32db4e9a82a584a7adf27573974f0d5346f717da059c2cab79"} Sep 10 11:59:05.534945 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:05.534547 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:05.553773 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:05.553726 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" podStartSLOduration=2.060892576 podStartE2EDuration="3.553708618s" totalImagesPullingTime="1.492816042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:59:05.551579026 +0000 UTC m=+453.430968463" watchObservedRunningTime="2026-09-10 11:59:05.553708618 +0000 UTC m=+453.433098050" Sep 10 11:59:07.161740 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:07.161709 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2"] Sep 10 11:59:07.165100 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:07.165085 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:07.169662 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:07.169643 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4mtmh\"" Sep 10 11:59:07.181937 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:07.181909 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2"] Sep 10 11:59:07.258508 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:07.258473 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4zj\" (UniqueName: \"kubernetes.io/projected/dc33e064-52ef-402e-bfc6-45425691be6d-kube-api-access-bx4zj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" (UID: \"dc33e064-52ef-402e-bfc6-45425691be6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:07.258661 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:07.258523 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc33e064-52ef-402e-bfc6-45425691be6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" (UID: \"dc33e064-52ef-402e-bfc6-45425691be6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:07.358957 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:07.358921 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4zj\" (UniqueName: \"kubernetes.io/projected/dc33e064-52ef-402e-bfc6-45425691be6d-kube-api-access-bx4zj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" (UID: \"dc33e064-52ef-402e-bfc6-45425691be6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:07.359121 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:07.358977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc33e064-52ef-402e-bfc6-45425691be6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" (UID: \"dc33e064-52ef-402e-bfc6-45425691be6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:07.359316 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:07.359300 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc33e064-52ef-402e-bfc6-45425691be6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" (UID: \"dc33e064-52ef-402e-bfc6-45425691be6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:07.368680 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:07.368658 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4zj\" (UniqueName: \"kubernetes.io/projected/dc33e064-52ef-402e-bfc6-45425691be6d-kube-api-access-bx4zj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" (UID: \"dc33e064-52ef-402e-bfc6-45425691be6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:07.474748 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:07.474722 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:07.756193 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:07.756103 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2"] Sep 10 11:59:07.759223 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:59:07.759188 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc33e064_52ef_402e_bfc6_45425691be6d.slice/crio-818dc550c72e4658fca558fcf3b68ce6bf2ea7e231678d59f4ef04d14aaccbbc WatchSource:0}: Error finding container 818dc550c72e4658fca558fcf3b68ce6bf2ea7e231678d59f4ef04d14aaccbbc: Status 404 returned error can't find the container with id 818dc550c72e4658fca558fcf3b68ce6bf2ea7e231678d59f4ef04d14aaccbbc Sep 10 11:59:08.551831 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:08.551789 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" event={"ID":"dc33e064-52ef-402e-bfc6-45425691be6d","Type":"ContainerStarted","Data":"818dc550c72e4658fca558fcf3b68ce6bf2ea7e231678d59f4ef04d14aaccbbc"} Sep 10 11:59:09.299107 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:09.299071 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-pkw6v"] Sep 10 11:59:09.302769 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:09.302743 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-pkw6v" Sep 10 11:59:09.305463 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:09.305428 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8dw78\"" Sep 10 11:59:09.322012 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:09.321988 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-pkw6v"] Sep 10 11:59:09.378006 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:09.377974 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns4k4\" (UniqueName: \"kubernetes.io/projected/7da9e8ee-1931-4828-9aba-4f17f98cbd65-kube-api-access-ns4k4\") pod \"authorino-operator-76b478bd78-pkw6v\" (UID: \"7da9e8ee-1931-4828-9aba-4f17f98cbd65\") " pod="kuadrant-system/authorino-operator-76b478bd78-pkw6v" Sep 10 11:59:09.479020 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:09.478984 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns4k4\" (UniqueName: \"kubernetes.io/projected/7da9e8ee-1931-4828-9aba-4f17f98cbd65-kube-api-access-ns4k4\") pod \"authorino-operator-76b478bd78-pkw6v\" (UID: \"7da9e8ee-1931-4828-9aba-4f17f98cbd65\") " pod="kuadrant-system/authorino-operator-76b478bd78-pkw6v" Sep 10 11:59:09.498224 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:09.498192 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns4k4\" (UniqueName: \"kubernetes.io/projected/7da9e8ee-1931-4828-9aba-4f17f98cbd65-kube-api-access-ns4k4\") pod \"authorino-operator-76b478bd78-pkw6v\" (UID: \"7da9e8ee-1931-4828-9aba-4f17f98cbd65\") " pod="kuadrant-system/authorino-operator-76b478bd78-pkw6v" Sep 10 11:59:09.616135 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:09.616043 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-pkw6v" Sep 10 11:59:09.901342 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:09.901254 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-pkw6v"] Sep 10 11:59:09.904914 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:59:09.904884 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da9e8ee_1931_4828_9aba_4f17f98cbd65.slice/crio-8a92bbe516dcd1a8f71a72a2d9f8789e2fd3d84443234bf40c6e6a920b2efc5e WatchSource:0}: Error finding container 8a92bbe516dcd1a8f71a72a2d9f8789e2fd3d84443234bf40c6e6a920b2efc5e: Status 404 returned error can't find the container with id 8a92bbe516dcd1a8f71a72a2d9f8789e2fd3d84443234bf40c6e6a920b2efc5e Sep 10 11:59:10.565137 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:10.565090 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-pkw6v" event={"ID":"7da9e8ee-1931-4828-9aba-4f17f98cbd65","Type":"ContainerStarted","Data":"8a92bbe516dcd1a8f71a72a2d9f8789e2fd3d84443234bf40c6e6a920b2efc5e"} Sep 10 11:59:11.573351 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:11.573315 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" event={"ID":"dc33e064-52ef-402e-bfc6-45425691be6d","Type":"ContainerStarted","Data":"5e6595eecb1efe942c27ca8d05262f696d1e7213a0ad2a3fe97b5637b634cc83"} Sep 10 11:59:11.573824 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:11.573466 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:11.596981 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:11.596934 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" podStartSLOduration=0.932848721 podStartE2EDuration="4.596918982s" totalImagesPullingTime="3.664070261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:59:11.594973446 +0000 UTC m=+459.474362879" watchObservedRunningTime="2026-09-10 11:59:11.596918982 +0000 UTC m=+459.476308413" Sep 10 11:59:12.580299 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:12.580253 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-pkw6v" event={"ID":"7da9e8ee-1931-4828-9aba-4f17f98cbd65","Type":"ContainerStarted","Data":"2590874a42ccdd858b13702aedb1b5667acf1091fd55bfb10385dfb8c17c0177"} Sep 10 11:59:12.580752 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:12.580472 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-pkw6v" Sep 10 11:59:12.599401 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:12.599355 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-pkw6v" podStartSLOduration=1.358928785 podStartE2EDuration="3.599342163s" totalImagesPullingTime="2.240413378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:59:12.599239403 +0000 UTC m=+460.478628837" watchObservedRunningTime="2026-09-10 11:59:12.599342163 +0000 UTC m=+460.478731595" Sep 10 11:59:14.530878 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:14.530848 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-5sn2n" Sep 10 11:59:16.541909 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:16.541881 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:22.581856 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:22.581824 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:23.587851 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:23.587820 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-pkw6v" Sep 10 11:59:24.939200 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:24.939163 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2"] Sep 10 11:59:24.939610 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:24.939382 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" containerName="manager" containerID="cri-o://5e6595eecb1efe942c27ca8d05262f696d1e7213a0ad2a3fe97b5637b634cc83" gracePeriod=2 Sep 10 11:59:24.958303 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:24.958279 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2"] Sep 10 11:59:24.965396 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:24.965365 2582 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dc33e064-52ef-402e-bfc6-45425691be6d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\": pods \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:24.984286 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:24.984257 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5"] Sep 10 11:59:24.984559 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:24.984501 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" containerName="manager" containerID="cri-o://c3ae0376e477cf32db4e9a82a584a7adf27573974f0d5346f717da059c2cab79" gracePeriod=2 Sep 10 11:59:24.986047 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:24.986023 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn"] Sep 10 11:59:24.986401 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:24.986388 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" containerName="manager" Sep 10 11:59:24.986460 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:24.986403 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" containerName="manager" Sep 10 11:59:24.986502 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:24.986482 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" containerName="manager" Sep 10 11:59:24.989610 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:24.989589 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" Sep 10 11:59:24.993846 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:24.993810 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:24.996179 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:24.996129 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5"] Sep 10 11:59:24.996285 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:24.996245 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9bpb5\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:25.014003 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.013972 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn"] Sep 10 11:59:25.034306 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.034277 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-z65zz"] Sep 10 11:59:25.034853 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.034829 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" containerName="manager" Sep 10 11:59:25.034853 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.034856 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" containerName="manager" Sep 10 11:59:25.035045 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.034970 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" containerName="manager" Sep 10 11:59:25.038299 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.038270 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:25.038299 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.038291 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-z65zz" Sep 10 11:59:25.043319 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.043291 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9bpb5\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:25.049063 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.049038 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9bpb5\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:25.052513 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.052379 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:25.054960 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.054937 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-z65zz"] Sep 10 11:59:25.112986 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.112954 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tppdx\" (UniqueName: \"kubernetes.io/projected/07c51ba4-9985-49a7-a923-c0f50c3cea28-kube-api-access-tppdx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wtfmn\" (UID: \"07c51ba4-9985-49a7-a923-c0f50c3cea28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" Sep 10 11:59:25.113135 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.112992 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zngr\" (UniqueName: \"kubernetes.io/projected/1f556bb6-6163-4685-afcf-d1151a4dbb9d-kube-api-access-7zngr\") pod \"limitador-operator-controller-manager-74495c69f8-z65zz\" (UID: \"1f556bb6-6163-4685-afcf-d1151a4dbb9d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-z65zz" Sep 10 11:59:25.113135 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.113095 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c51ba4-9985-49a7-a923-c0f50c3cea28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wtfmn\" (UID: \"07c51ba4-9985-49a7-a923-c0f50c3cea28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" Sep 10 11:59:25.174496 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.174466 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm"] Sep 10 11:59:25.178426 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.178401 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" Sep 10 11:59:25.181307 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.181272 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:25.202457 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.202367 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm"] Sep 10 11:59:25.213982 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.213957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c51ba4-9985-49a7-a923-c0f50c3cea28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wtfmn\" (UID: \"07c51ba4-9985-49a7-a923-c0f50c3cea28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" Sep 10 11:59:25.214108 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.214051 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tppdx\" (UniqueName: \"kubernetes.io/projected/07c51ba4-9985-49a7-a923-c0f50c3cea28-kube-api-access-tppdx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wtfmn\" (UID: \"07c51ba4-9985-49a7-a923-c0f50c3cea28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" Sep 10 11:59:25.214108 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.214082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zngr\" (UniqueName: \"kubernetes.io/projected/1f556bb6-6163-4685-afcf-d1151a4dbb9d-kube-api-access-7zngr\") pod \"limitador-operator-controller-manager-74495c69f8-z65zz\" (UID: \"1f556bb6-6163-4685-afcf-d1151a4dbb9d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-z65zz" Sep 10 11:59:25.214344 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.214323 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c51ba4-9985-49a7-a923-c0f50c3cea28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wtfmn\" (UID: \"07c51ba4-9985-49a7-a923-c0f50c3cea28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" Sep 10 11:59:25.230002 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.229968 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9bpb5\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:25.230111 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.230002 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zngr\" (UniqueName: \"kubernetes.io/projected/1f556bb6-6163-4685-afcf-d1151a4dbb9d-kube-api-access-7zngr\") pod \"limitador-operator-controller-manager-74495c69f8-z65zz\" (UID: \"1f556bb6-6163-4685-afcf-d1151a4dbb9d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-z65zz" Sep 10 11:59:25.231786 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.231765 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tppdx\" (UniqueName: \"kubernetes.io/projected/07c51ba4-9985-49a7-a923-c0f50c3cea28-kube-api-access-tppdx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wtfmn\" (UID: \"07c51ba4-9985-49a7-a923-c0f50c3cea28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" Sep 10 11:59:25.234479 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.234457 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:25.237007 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.236991 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:25.237276 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.237258 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9bpb5\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:25.241571 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.241549 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:25.247900 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.247880 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9bpb5\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:25.250203 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.250185 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:25.314829 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.314800 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx4zj\" (UniqueName: \"kubernetes.io/projected/dc33e064-52ef-402e-bfc6-45425691be6d-kube-api-access-bx4zj\") pod \"dc33e064-52ef-402e-bfc6-45425691be6d\" (UID: \"dc33e064-52ef-402e-bfc6-45425691be6d\") " Sep 10 11:59:25.314959 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.314836 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc33e064-52ef-402e-bfc6-45425691be6d-extensions-socket-volume\") pod \"dc33e064-52ef-402e-bfc6-45425691be6d\" (UID: \"dc33e064-52ef-402e-bfc6-45425691be6d\") " Sep 10 11:59:25.314959 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.314906 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc74q\" (UniqueName: \"kubernetes.io/projected/3c80af4f-10ac-4f5c-8988-a65ca97371b9-kube-api-access-tc74q\") pod \"3c80af4f-10ac-4f5c-8988-a65ca97371b9\" (UID: \"3c80af4f-10ac-4f5c-8988-a65ca97371b9\") " Sep 10 11:59:25.315030 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.314989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b61d2f4-ecfc-465a-a2f0-5849df3e0608-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-t89fm\" (UID: \"5b61d2f4-ecfc-465a-a2f0-5849df3e0608\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" Sep 10 11:59:25.315067 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.315056 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kdhk\" (UniqueName: \"kubernetes.io/projected/5b61d2f4-ecfc-465a-a2f0-5849df3e0608-kube-api-access-7kdhk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-t89fm\" (UID: \"5b61d2f4-ecfc-465a-a2f0-5849df3e0608\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" Sep 10 11:59:25.315428 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.315392 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc33e064-52ef-402e-bfc6-45425691be6d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dc33e064-52ef-402e-bfc6-45425691be6d" (UID: "dc33e064-52ef-402e-bfc6-45425691be6d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 11:59:25.317229 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.317208 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc33e064-52ef-402e-bfc6-45425691be6d-kube-api-access-bx4zj" (OuterVolumeSpecName: "kube-api-access-bx4zj") pod "dc33e064-52ef-402e-bfc6-45425691be6d" (UID: "dc33e064-52ef-402e-bfc6-45425691be6d"). InnerVolumeSpecName "kube-api-access-bx4zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 11:59:25.317320 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.317246 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c80af4f-10ac-4f5c-8988-a65ca97371b9-kube-api-access-tc74q" (OuterVolumeSpecName: "kube-api-access-tc74q") pod "3c80af4f-10ac-4f5c-8988-a65ca97371b9" (UID: "3c80af4f-10ac-4f5c-8988-a65ca97371b9"). InnerVolumeSpecName "kube-api-access-tc74q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 11:59:25.369139 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.369095 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" Sep 10 11:59:25.374105 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.374081 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-z65zz" Sep 10 11:59:25.416797 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.416310 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kdhk\" (UniqueName: \"kubernetes.io/projected/5b61d2f4-ecfc-465a-a2f0-5849df3e0608-kube-api-access-7kdhk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-t89fm\" (UID: \"5b61d2f4-ecfc-465a-a2f0-5849df3e0608\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" Sep 10 11:59:25.416797 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.416370 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b61d2f4-ecfc-465a-a2f0-5849df3e0608-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-t89fm\" (UID: \"5b61d2f4-ecfc-465a-a2f0-5849df3e0608\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" Sep 10 11:59:25.416797 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.416422 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc74q\" (UniqueName: \"kubernetes.io/projected/3c80af4f-10ac-4f5c-8988-a65ca97371b9-kube-api-access-tc74q\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:59:25.416797 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.416434 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx4zj\" (UniqueName: \"kubernetes.io/projected/dc33e064-52ef-402e-bfc6-45425691be6d-kube-api-access-bx4zj\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:59:25.416797 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.416444 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc33e064-52ef-402e-bfc6-45425691be6d-extensions-socket-volume\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:59:25.416797 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.416746 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b61d2f4-ecfc-465a-a2f0-5849df3e0608-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-t89fm\" (UID: \"5b61d2f4-ecfc-465a-a2f0-5849df3e0608\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" Sep 10 11:59:25.431493 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.431462 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kdhk\" (UniqueName: \"kubernetes.io/projected/5b61d2f4-ecfc-465a-a2f0-5849df3e0608-kube-api-access-7kdhk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-t89fm\" (UID: \"5b61d2f4-ecfc-465a-a2f0-5849df3e0608\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" Sep 10 11:59:25.489347 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.489267 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" Sep 10 11:59:25.641819 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.641787 2582 generic.go:356] "Generic (PLEG): container finished" podID="dc33e064-52ef-402e-bfc6-45425691be6d" containerID="5e6595eecb1efe942c27ca8d05262f696d1e7213a0ad2a3fe97b5637b634cc83" exitCode=0 Sep 10 11:59:25.641974 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.641834 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:25.641974 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.641893 2582 scope.go:117] "RemoveContainer" containerID="5e6595eecb1efe942c27ca8d05262f696d1e7213a0ad2a3fe97b5637b634cc83" Sep 10 11:59:25.643375 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.643350 2582 generic.go:356] "Generic (PLEG): container finished" podID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" containerID="c3ae0376e477cf32db4e9a82a584a7adf27573974f0d5346f717da059c2cab79" exitCode=0 Sep 10 11:59:25.643529 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.643403 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:25.647205 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.647178 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9bpb5\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:25.650025 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.650001 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:25.650895 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.650880 2582 scope.go:117] "RemoveContainer" containerID="5e6595eecb1efe942c27ca8d05262f696d1e7213a0ad2a3fe97b5637b634cc83" Sep 10 11:59:25.651262 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.651243 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6595eecb1efe942c27ca8d05262f696d1e7213a0ad2a3fe97b5637b634cc83\": container with ID starting with 5e6595eecb1efe942c27ca8d05262f696d1e7213a0ad2a3fe97b5637b634cc83 not found: ID does not exist" containerID="5e6595eecb1efe942c27ca8d05262f696d1e7213a0ad2a3fe97b5637b634cc83" Sep 10 11:59:25.651333 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.651268 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6595eecb1efe942c27ca8d05262f696d1e7213a0ad2a3fe97b5637b634cc83"} err="failed to get container status \"5e6595eecb1efe942c27ca8d05262f696d1e7213a0ad2a3fe97b5637b634cc83\": rpc error: code = NotFound desc = could not find container \"5e6595eecb1efe942c27ca8d05262f696d1e7213a0ad2a3fe97b5637b634cc83\": container with ID starting with 5e6595eecb1efe942c27ca8d05262f696d1e7213a0ad2a3fe97b5637b634cc83 not found: ID does not exist" Sep 10 11:59:25.651333 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.651284 2582 scope.go:117] "RemoveContainer" containerID="c3ae0376e477cf32db4e9a82a584a7adf27573974f0d5346f717da059c2cab79" Sep 10 11:59:25.652877 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.652851 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9bpb5\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:25.658702 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.658687 2582 scope.go:117] "RemoveContainer" containerID="c3ae0376e477cf32db4e9a82a584a7adf27573974f0d5346f717da059c2cab79" Sep 10 11:59:25.658947 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.658930 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ae0376e477cf32db4e9a82a584a7adf27573974f0d5346f717da059c2cab79\": container with ID starting with c3ae0376e477cf32db4e9a82a584a7adf27573974f0d5346f717da059c2cab79 not found: ID does not exist" containerID="c3ae0376e477cf32db4e9a82a584a7adf27573974f0d5346f717da059c2cab79" Sep 10 11:59:25.658987 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.658955 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ae0376e477cf32db4e9a82a584a7adf27573974f0d5346f717da059c2cab79"} err="failed to get container status \"c3ae0376e477cf32db4e9a82a584a7adf27573974f0d5346f717da059c2cab79\": rpc error: code = NotFound desc = could not find container \"c3ae0376e477cf32db4e9a82a584a7adf27573974f0d5346f717da059c2cab79\": container with ID starting with c3ae0376e477cf32db4e9a82a584a7adf27573974f0d5346f717da059c2cab79 not found: ID does not exist" Sep 10 11:59:25.660415 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.660397 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:25.663243 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.663218 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9bpb5\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9bpb5" Sep 10 11:59:25.671898 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:25.671871 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rkpg2\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rkpg2" Sep 10 11:59:25.693814 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.693791 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-z65zz"] Sep 10 11:59:25.696358 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:59:25.696317 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f556bb6_6163_4685_afcf_d1151a4dbb9d.slice/crio-29c9466755f28cedc727b29a562400aa644e2a4c61f5227fc79273d5e02bd288 WatchSource:0}: Error finding container 29c9466755f28cedc727b29a562400aa644e2a4c61f5227fc79273d5e02bd288: Status 404 returned error can't find the container with id 29c9466755f28cedc727b29a562400aa644e2a4c61f5227fc79273d5e02bd288 Sep 10 11:59:25.698924 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.698905 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn"] Sep 10 11:59:25.700296 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:59:25.700271 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c51ba4_9985_49a7_a923_c0f50c3cea28.slice/crio-9086adeb895fd5223c6348bdf710634f2b63f25674908f24cc45994721a93f18 WatchSource:0}: Error finding container 9086adeb895fd5223c6348bdf710634f2b63f25674908f24cc45994721a93f18: Status 404 returned error can't find the container with id 9086adeb895fd5223c6348bdf710634f2b63f25674908f24cc45994721a93f18 Sep 10 11:59:25.785549 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:25.785525 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm"] Sep 10 11:59:25.788106 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:59:25.788085 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b61d2f4_ecfc_465a_a2f0_5849df3e0608.slice/crio-9a787fa9fd278bd0f135edb1a34fa0f788c111b100b905ca44cb178282b82183 WatchSource:0}: Error finding container 9a787fa9fd278bd0f135edb1a34fa0f788c111b100b905ca44cb178282b82183: Status 404 returned error can't find the container with id 9a787fa9fd278bd0f135edb1a34fa0f788c111b100b905ca44cb178282b82183 Sep 10 11:59:26.630490 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.630458 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c80af4f-10ac-4f5c-8988-a65ca97371b9" path="/var/lib/kubelet/pods/3c80af4f-10ac-4f5c-8988-a65ca97371b9/volumes" Sep 10 11:59:26.630826 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.630789 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc33e064-52ef-402e-bfc6-45425691be6d" path="/var/lib/kubelet/pods/dc33e064-52ef-402e-bfc6-45425691be6d/volumes" Sep 10 11:59:26.654427 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.654394 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" event={"ID":"07c51ba4-9985-49a7-a923-c0f50c3cea28","Type":"ContainerStarted","Data":"ca4b1c52fc97a0c812a27a84f90e2a0e4a786fcf6447b6cfcd2483d067d40cd4"} Sep 10 11:59:26.654584 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.654491 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" Sep 10 11:59:26.654584 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.654522 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" event={"ID":"07c51ba4-9985-49a7-a923-c0f50c3cea28","Type":"ContainerStarted","Data":"9086adeb895fd5223c6348bdf710634f2b63f25674908f24cc45994721a93f18"} Sep 10 11:59:26.657535 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.657501 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" event={"ID":"5b61d2f4-ecfc-465a-a2f0-5849df3e0608","Type":"ContainerStarted","Data":"6b5faac52e41619ad76d73d7a35f88f3c466b1afd4f56e91f2d14b9bb3d1a9e6"} Sep 10 11:59:26.657639 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.657548 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" event={"ID":"5b61d2f4-ecfc-465a-a2f0-5849df3e0608","Type":"ContainerStarted","Data":"9a787fa9fd278bd0f135edb1a34fa0f788c111b100b905ca44cb178282b82183"} Sep 10 11:59:26.657639 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.657616 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" Sep 10 11:59:26.660110 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.660079 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-z65zz" event={"ID":"1f556bb6-6163-4685-afcf-d1151a4dbb9d","Type":"ContainerStarted","Data":"b114ac320df63d2d3c6f84a46ffd487e6b91712b53144798b886c4881403c5e1"} Sep 10 11:59:26.660257 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.660116 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-z65zz" event={"ID":"1f556bb6-6163-4685-afcf-d1151a4dbb9d","Type":"ContainerStarted","Data":"29c9466755f28cedc727b29a562400aa644e2a4c61f5227fc79273d5e02bd288"} Sep 10 11:59:26.660257 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.660195 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-z65zz" Sep 10 11:59:26.679827 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.679784 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" podStartSLOduration=2.6797735019999998 podStartE2EDuration="2.679773502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:59:26.679596262 +0000 UTC m=+474.558985695" watchObservedRunningTime="2026-09-10 11:59:26.679773502 +0000 UTC m=+474.559162936" Sep 10 11:59:26.707814 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.707769 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-z65zz" podStartSLOduration=2.707755155 podStartE2EDuration="2.707755155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:59:26.706837247 +0000 UTC m=+474.586226679" watchObservedRunningTime="2026-09-10 11:59:26.707755155 +0000 UTC m=+474.587144587" Sep 10 11:59:26.736542 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:26.736501 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" podStartSLOduration=1.736487462 podStartE2EDuration="1.736487462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:59:26.734800121 +0000 UTC m=+474.614189553" watchObservedRunningTime="2026-09-10 11:59:26.736487462 +0000 UTC m=+474.615876895" Sep 10 11:59:37.666806 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:37.666776 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" Sep 10 11:59:37.667283 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:37.666859 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" Sep 10 11:59:37.667283 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:37.667017 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-z65zz" Sep 10 11:59:37.777543 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:37.777509 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn"] Sep 10 11:59:37.777757 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:37.777735 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" podUID="07c51ba4-9985-49a7-a923-c0f50c3cea28" containerName="manager" containerID="cri-o://ca4b1c52fc97a0c812a27a84f90e2a0e4a786fcf6447b6cfcd2483d067d40cd4" gracePeriod=10 Sep 10 11:59:38.024858 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.024836 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" Sep 10 11:59:38.132316 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.132283 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c51ba4-9985-49a7-a923-c0f50c3cea28-extensions-socket-volume\") pod \"07c51ba4-9985-49a7-a923-c0f50c3cea28\" (UID: \"07c51ba4-9985-49a7-a923-c0f50c3cea28\") " Sep 10 11:59:38.132514 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.132377 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tppdx\" (UniqueName: \"kubernetes.io/projected/07c51ba4-9985-49a7-a923-c0f50c3cea28-kube-api-access-tppdx\") pod \"07c51ba4-9985-49a7-a923-c0f50c3cea28\" (UID: \"07c51ba4-9985-49a7-a923-c0f50c3cea28\") " Sep 10 11:59:38.132705 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.132678 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07c51ba4-9985-49a7-a923-c0f50c3cea28-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "07c51ba4-9985-49a7-a923-c0f50c3cea28" (UID: "07c51ba4-9985-49a7-a923-c0f50c3cea28"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 11:59:38.134562 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.134530 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c51ba4-9985-49a7-a923-c0f50c3cea28-kube-api-access-tppdx" (OuterVolumeSpecName: "kube-api-access-tppdx") pod "07c51ba4-9985-49a7-a923-c0f50c3cea28" (UID: "07c51ba4-9985-49a7-a923-c0f50c3cea28"). InnerVolumeSpecName "kube-api-access-tppdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 11:59:38.233827 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.233748 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07c51ba4-9985-49a7-a923-c0f50c3cea28-extensions-socket-volume\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:59:38.233827 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.233773 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tppdx\" (UniqueName: \"kubernetes.io/projected/07c51ba4-9985-49a7-a923-c0f50c3cea28-kube-api-access-tppdx\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 11:59:38.717663 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.717629 2582 generic.go:356] "Generic (PLEG): container finished" podID="07c51ba4-9985-49a7-a923-c0f50c3cea28" containerID="ca4b1c52fc97a0c812a27a84f90e2a0e4a786fcf6447b6cfcd2483d067d40cd4" exitCode=0 Sep 10 11:59:38.718042 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.717676 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" Sep 10 11:59:38.718042 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.717722 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" event={"ID":"07c51ba4-9985-49a7-a923-c0f50c3cea28","Type":"ContainerDied","Data":"ca4b1c52fc97a0c812a27a84f90e2a0e4a786fcf6447b6cfcd2483d067d40cd4"} Sep 10 11:59:38.718042 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.717774 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn" event={"ID":"07c51ba4-9985-49a7-a923-c0f50c3cea28","Type":"ContainerDied","Data":"9086adeb895fd5223c6348bdf710634f2b63f25674908f24cc45994721a93f18"} Sep 10 11:59:38.718042 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.717805 2582 scope.go:117] "RemoveContainer" containerID="ca4b1c52fc97a0c812a27a84f90e2a0e4a786fcf6447b6cfcd2483d067d40cd4" Sep 10 11:59:38.727941 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.727917 2582 scope.go:117] "RemoveContainer" containerID="ca4b1c52fc97a0c812a27a84f90e2a0e4a786fcf6447b6cfcd2483d067d40cd4" Sep 10 11:59:38.728351 ip-10-0-136-197 kubenswrapper[2582]: E0910 11:59:38.728331 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4b1c52fc97a0c812a27a84f90e2a0e4a786fcf6447b6cfcd2483d067d40cd4\": container with ID starting with ca4b1c52fc97a0c812a27a84f90e2a0e4a786fcf6447b6cfcd2483d067d40cd4 not found: ID does not exist" containerID="ca4b1c52fc97a0c812a27a84f90e2a0e4a786fcf6447b6cfcd2483d067d40cd4" Sep 10 11:59:38.728470 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.728359 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4b1c52fc97a0c812a27a84f90e2a0e4a786fcf6447b6cfcd2483d067d40cd4"} err="failed to get container status \"ca4b1c52fc97a0c812a27a84f90e2a0e4a786fcf6447b6cfcd2483d067d40cd4\": rpc error: code = NotFound desc = could not find container \"ca4b1c52fc97a0c812a27a84f90e2a0e4a786fcf6447b6cfcd2483d067d40cd4\": container with ID starting with ca4b1c52fc97a0c812a27a84f90e2a0e4a786fcf6447b6cfcd2483d067d40cd4 not found: ID does not exist" Sep 10 11:59:38.748539 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.748514 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn"] Sep 10 11:59:38.752740 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:38.752718 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtfmn"] Sep 10 11:59:40.629735 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:40.629698 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07c51ba4-9985-49a7-a923-c0f50c3cea28" path="/var/lib/kubelet/pods/07c51ba4-9985-49a7-a923-c0f50c3cea28/volumes" Sep 10 11:59:54.196704 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.196672 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf"] Sep 10 11:59:54.197191 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.197019 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c51ba4-9985-49a7-a923-c0f50c3cea28" containerName="manager" Sep 10 11:59:54.197191 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.197029 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c51ba4-9985-49a7-a923-c0f50c3cea28" containerName="manager" Sep 10 11:59:54.197191 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.197087 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c51ba4-9985-49a7-a923-c0f50c3cea28" containerName="manager" Sep 10 11:59:54.199504 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.199485 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.202237 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.202213 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 11:59:54.202596 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.202571 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ksk6j\"" Sep 10 11:59:54.212560 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.212537 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf"] Sep 10 11:59:54.260356 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.260309 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.260356 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.260362 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.260576 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.260382 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.260576 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.260401 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.260576 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.260418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.260576 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.260446 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.260576 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.260545 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.260576 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.260571 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.261069 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.260633 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l48w\" (UniqueName: \"kubernetes.io/projected/13ee911d-d862-4efa-b0c0-7bf76dbabaab-kube-api-access-4l48w\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.361736 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.361694 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l48w\" (UniqueName: \"kubernetes.io/projected/13ee911d-d862-4efa-b0c0-7bf76dbabaab-kube-api-access-4l48w\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.361736 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.361740 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.361989 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.361868 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.361989 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.361910 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.361989 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.361930 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.361989 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.361950 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.361989 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.361984 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.362288 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.362019 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.362288 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.362043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.362288 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.362166 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.362456 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.362434 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.362518 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.362464 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.362518 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.362477 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.362679 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.362660 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.364515 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.364491 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.364644 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.364626 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.370024 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.369999 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13ee911d-d862-4efa-b0c0-7bf76dbabaab-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.370169 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.370125 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l48w\" (UniqueName: \"kubernetes.io/projected/13ee911d-d862-4efa-b0c0-7bf76dbabaab-kube-api-access-4l48w\") pod \"maas-default-gateway-openshift-default-5ccb968c66-thwsf\" (UID: \"13ee911d-d862-4efa-b0c0-7bf76dbabaab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.510465 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.510386 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:54.787399 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:54.787375 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf"] Sep 10 11:59:54.789772 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:59:54.789706 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ee911d_d862_4efa_b0c0_7bf76dbabaab.slice/crio-af54f6a4ccca0a33ca194a1374ae6eb3b89d38446b0259e1f4ee347d46030604 WatchSource:0}: Error finding container af54f6a4ccca0a33ca194a1374ae6eb3b89d38446b0259e1f4ee347d46030604: Status 404 returned error can't find the container with id af54f6a4ccca0a33ca194a1374ae6eb3b89d38446b0259e1f4ee347d46030604 Sep 10 11:59:55.800559 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:55.800520 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" event={"ID":"13ee911d-d862-4efa-b0c0-7bf76dbabaab","Type":"ContainerStarted","Data":"af54f6a4ccca0a33ca194a1374ae6eb3b89d38446b0259e1f4ee347d46030604"} Sep 10 11:59:57.428489 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:57.428452 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 11:59:57.428856 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:57.428536 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 11:59:57.428856 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:57.428586 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 11:59:57.811461 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:57.811424 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" event={"ID":"13ee911d-d862-4efa-b0c0-7bf76dbabaab","Type":"ContainerStarted","Data":"f594d466f62bbf9daf6a37a2b65748d5f7419811bf7886f9b845a1b7e399b60e"} Sep 10 11:59:57.836457 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:57.836409 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" podStartSLOduration=1.199870947 podStartE2EDuration="3.836394208s" totalImagesPullingTime="2.636523261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:59:57.834113266 +0000 UTC m=+505.713502698" watchObservedRunningTime="2026-09-10 11:59:57.836394208 +0000 UTC m=+505.715783640" Sep 10 11:59:58.510762 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:58.510726 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:58.515448 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:58.515425 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:58.816984 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:58.816912 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:58.817969 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:58.817950 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 11:59:59.229039 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.229008 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 11:59:59.232461 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.232446 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-54pt4" Sep 10 11:59:59.235296 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.235265 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 10 11:59:59.235540 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.235525 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cvsxm\"" Sep 10 11:59:59.251749 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.251723 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 11:59:59.310870 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.310842 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5grn\" (UniqueName: \"kubernetes.io/projected/35563073-52de-46b6-913e-da70e41d9355-kube-api-access-r5grn\") pod \"limitador-limitador-84994f7ffc-54pt4\" (UID: \"35563073-52de-46b6-913e-da70e41d9355\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-54pt4" Sep 10 11:59:59.310997 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.310902 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/35563073-52de-46b6-913e-da70e41d9355-config-file\") pod \"limitador-limitador-84994f7ffc-54pt4\" (UID: \"35563073-52de-46b6-913e-da70e41d9355\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-54pt4" Sep 10 11:59:59.328634 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.328600 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 11:59:59.411790 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.411761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5grn\" (UniqueName: \"kubernetes.io/projected/35563073-52de-46b6-913e-da70e41d9355-kube-api-access-r5grn\") pod \"limitador-limitador-84994f7ffc-54pt4\" (UID: \"35563073-52de-46b6-913e-da70e41d9355\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-54pt4" Sep 10 11:59:59.411975 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.411809 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/35563073-52de-46b6-913e-da70e41d9355-config-file\") pod \"limitador-limitador-84994f7ffc-54pt4\" (UID: \"35563073-52de-46b6-913e-da70e41d9355\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-54pt4" Sep 10 11:59:59.412448 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.412430 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/35563073-52de-46b6-913e-da70e41d9355-config-file\") pod \"limitador-limitador-84994f7ffc-54pt4\" (UID: \"35563073-52de-46b6-913e-da70e41d9355\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-54pt4" Sep 10 11:59:59.424997 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.424974 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5grn\" (UniqueName: \"kubernetes.io/projected/35563073-52de-46b6-913e-da70e41d9355-kube-api-access-r5grn\") pod \"limitador-limitador-84994f7ffc-54pt4\" (UID: \"35563073-52de-46b6-913e-da70e41d9355\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-54pt4" Sep 10 11:59:59.543857 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.543772 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-54pt4" Sep 10 11:59:59.736760 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.736731 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gwzk5"] Sep 10 11:59:59.741290 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.741268 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gwzk5" Sep 10 11:59:59.744966 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.744946 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-m6mmm\"" Sep 10 11:59:59.749615 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.749595 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gwzk5"] Sep 10 11:59:59.815372 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.815283 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln8fc\" (UniqueName: \"kubernetes.io/projected/9cd321a7-4513-4ae4-8db7-1b664871f133-kube-api-access-ln8fc\") pod \"authorino-7bcfd8f9cd-gwzk5\" (UID: \"9cd321a7-4513-4ae4-8db7-1b664871f133\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gwzk5" Sep 10 11:59:59.817109 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.817080 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 11:59:59.819405 ip-10-0-136-197 kubenswrapper[2582]: W0910 11:59:59.819384 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35563073_52de_46b6_913e_da70e41d9355.slice/crio-320fd9fb565bade7e8830b9e16a87eee33ee4d28b0b0730dee8b2104d0995612 WatchSource:0}: Error finding container 320fd9fb565bade7e8830b9e16a87eee33ee4d28b0b0730dee8b2104d0995612: Status 404 returned error can't find the container with id 320fd9fb565bade7e8830b9e16a87eee33ee4d28b0b0730dee8b2104d0995612 Sep 10 11:59:59.916593 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.916560 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln8fc\" (UniqueName: \"kubernetes.io/projected/9cd321a7-4513-4ae4-8db7-1b664871f133-kube-api-access-ln8fc\") pod \"authorino-7bcfd8f9cd-gwzk5\" (UID: \"9cd321a7-4513-4ae4-8db7-1b664871f133\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gwzk5" Sep 10 11:59:59.925056 ip-10-0-136-197 kubenswrapper[2582]: I0910 11:59:59.925027 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln8fc\" (UniqueName: \"kubernetes.io/projected/9cd321a7-4513-4ae4-8db7-1b664871f133-kube-api-access-ln8fc\") pod \"authorino-7bcfd8f9cd-gwzk5\" (UID: \"9cd321a7-4513-4ae4-8db7-1b664871f133\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gwzk5" Sep 10 12:00:00.051163 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:00.051113 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gwzk5" Sep 10 12:00:00.321658 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:00.321627 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gwzk5"] Sep 10 12:00:00.325947 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:00:00.325915 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cd321a7_4513_4ae4_8db7_1b664871f133.slice/crio-fe1c46bb107be0d4b60d21085d6c058ada793728ffd8123232b27ed7e654d8e8 WatchSource:0}: Error finding container fe1c46bb107be0d4b60d21085d6c058ada793728ffd8123232b27ed7e654d8e8: Status 404 returned error can't find the container with id fe1c46bb107be0d4b60d21085d6c058ada793728ffd8123232b27ed7e654d8e8 Sep 10 12:00:00.828913 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:00.828878 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-54pt4" event={"ID":"35563073-52de-46b6-913e-da70e41d9355","Type":"ContainerStarted","Data":"320fd9fb565bade7e8830b9e16a87eee33ee4d28b0b0730dee8b2104d0995612"} Sep 10 12:00:00.830020 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:00.829997 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gwzk5" event={"ID":"9cd321a7-4513-4ae4-8db7-1b664871f133","Type":"ContainerStarted","Data":"fe1c46bb107be0d4b60d21085d6c058ada793728ffd8123232b27ed7e654d8e8"} Sep 10 12:00:01.837930 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:01.837894 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-54pt4" event={"ID":"35563073-52de-46b6-913e-da70e41d9355","Type":"ContainerStarted","Data":"e603ee756341dee65196aa618ce24b5fb2eea06b729b2d29df0584603558d98d"} Sep 10 12:00:01.838472 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:01.838042 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-54pt4" Sep 10 12:00:01.854981 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:01.854935 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-54pt4" podStartSLOduration=1.439705749 podStartE2EDuration="2.85491788s" totalImagesPullingTime="1.415212131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:00:01.854025776 +0000 UTC m=+509.733415210" watchObservedRunningTime="2026-09-10 12:00:01.85491788 +0000 UTC m=+509.734307312" Sep 10 12:00:02.845836 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:02.845800 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gwzk5" event={"ID":"9cd321a7-4513-4ae4-8db7-1b664871f133","Type":"ContainerStarted","Data":"5b549c85460618708501c9f8cc9e34f85fb915d909529c64c89b6691a64bb93f"} Sep 10 12:00:02.862298 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:02.862245 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-gwzk5" podStartSLOduration=1.489800392 podStartE2EDuration="3.862230223s" totalImagesPullingTime="2.372429831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:00:02.862053217 +0000 UTC m=+510.741442651" watchObservedRunningTime="2026-09-10 12:00:02.862230223 +0000 UTC m=+510.741619655" Sep 10 12:00:04.263286 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:04.263251 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gwzk5"] Sep 10 12:00:04.855596 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:04.855554 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-gwzk5" podUID="9cd321a7-4513-4ae4-8db7-1b664871f133" containerName="authorino" containerID="cri-o://5b549c85460618708501c9f8cc9e34f85fb915d909529c64c89b6691a64bb93f" gracePeriod=30 Sep 10 12:00:05.104167 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:05.104117 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gwzk5" Sep 10 12:00:05.163690 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:05.163603 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln8fc\" (UniqueName: \"kubernetes.io/projected/9cd321a7-4513-4ae4-8db7-1b664871f133-kube-api-access-ln8fc\") pod \"9cd321a7-4513-4ae4-8db7-1b664871f133\" (UID: \"9cd321a7-4513-4ae4-8db7-1b664871f133\") " Sep 10 12:00:05.165749 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:05.165718 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd321a7-4513-4ae4-8db7-1b664871f133-kube-api-access-ln8fc" (OuterVolumeSpecName: "kube-api-access-ln8fc") pod "9cd321a7-4513-4ae4-8db7-1b664871f133" (UID: "9cd321a7-4513-4ae4-8db7-1b664871f133"). InnerVolumeSpecName "kube-api-access-ln8fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:00:05.265047 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:05.265017 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln8fc\" (UniqueName: \"kubernetes.io/projected/9cd321a7-4513-4ae4-8db7-1b664871f133-kube-api-access-ln8fc\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:00:05.861864 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:05.861827 2582 generic.go:356] "Generic (PLEG): container finished" podID="9cd321a7-4513-4ae4-8db7-1b664871f133" containerID="5b549c85460618708501c9f8cc9e34f85fb915d909529c64c89b6691a64bb93f" exitCode=0 Sep 10 12:00:05.862020 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:05.861878 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gwzk5" Sep 10 12:00:05.862020 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:05.861912 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gwzk5" event={"ID":"9cd321a7-4513-4ae4-8db7-1b664871f133","Type":"ContainerDied","Data":"5b549c85460618708501c9f8cc9e34f85fb915d909529c64c89b6691a64bb93f"} Sep 10 12:00:05.862020 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:05.861952 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gwzk5" event={"ID":"9cd321a7-4513-4ae4-8db7-1b664871f133","Type":"ContainerDied","Data":"fe1c46bb107be0d4b60d21085d6c058ada793728ffd8123232b27ed7e654d8e8"} Sep 10 12:00:05.862020 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:05.861968 2582 scope.go:117] "RemoveContainer" containerID="5b549c85460618708501c9f8cc9e34f85fb915d909529c64c89b6691a64bb93f" Sep 10 12:00:05.870826 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:05.870812 2582 scope.go:117] "RemoveContainer" containerID="5b549c85460618708501c9f8cc9e34f85fb915d909529c64c89b6691a64bb93f" Sep 10 12:00:05.871080 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:00:05.871060 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b549c85460618708501c9f8cc9e34f85fb915d909529c64c89b6691a64bb93f\": container with ID starting with 5b549c85460618708501c9f8cc9e34f85fb915d909529c64c89b6691a64bb93f not found: ID does not exist" containerID="5b549c85460618708501c9f8cc9e34f85fb915d909529c64c89b6691a64bb93f" Sep 10 12:00:05.871117 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:05.871089 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b549c85460618708501c9f8cc9e34f85fb915d909529c64c89b6691a64bb93f"} err="failed to get container status \"5b549c85460618708501c9f8cc9e34f85fb915d909529c64c89b6691a64bb93f\": rpc error: code = NotFound desc = could not find container \"5b549c85460618708501c9f8cc9e34f85fb915d909529c64c89b6691a64bb93f\": container with ID starting with 5b549c85460618708501c9f8cc9e34f85fb915d909529c64c89b6691a64bb93f not found: ID does not exist" Sep 10 12:00:05.881952 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:05.881929 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gwzk5"] Sep 10 12:00:05.885920 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:05.885903 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gwzk5"] Sep 10 12:00:06.629120 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:06.629092 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd321a7-4513-4ae4-8db7-1b664871f133" path="/var/lib/kubelet/pods/9cd321a7-4513-4ae4-8db7-1b664871f133/volumes" Sep 10 12:00:12.847690 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:00:12.847658 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-54pt4" Sep 10 12:01:17.269177 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:17.269123 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:01:26.270646 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:26.270616 2582 generic.go:356] "Generic (PLEG): container finished" podID="13ee911d-d862-4efa-b0c0-7bf76dbabaab" containerID="f594d466f62bbf9daf6a37a2b65748d5f7419811bf7886f9b845a1b7e399b60e" exitCode=0 Sep 10 12:01:26.271059 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:26.270700 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" event={"ID":"13ee911d-d862-4efa-b0c0-7bf76dbabaab","Type":"ContainerDied","Data":"f594d466f62bbf9daf6a37a2b65748d5f7419811bf7886f9b845a1b7e399b60e"} Sep 10 12:01:26.271100 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:26.271059 2582 scope.go:117] "RemoveContainer" containerID="f594d466f62bbf9daf6a37a2b65748d5f7419811bf7886f9b845a1b7e399b60e" Sep 10 12:01:26.272044 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:26.272017 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:01:26.272097 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:26.272081 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:01:26.272131 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:26.272113 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:01:27.277273 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:27.277237 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" event={"ID":"13ee911d-d862-4efa-b0c0-7bf76dbabaab","Type":"ContainerStarted","Data":"78a6c5ef4c61e82ed0e5c226f4d8c787bd3948f680e21b2aeac0d2b1e7f31c2a"} Sep 10 12:01:27.512015 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:27.511977 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 12:01:27.512251 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:27.512114 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-thwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 12:01:27.512251 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:27.512178 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" podUID="13ee911d-d862-4efa-b0c0-7bf76dbabaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 12:01:28.511110 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:28.511064 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-thwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 12:01:28.511615 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:28.511134 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" podUID="13ee911d-d862-4efa-b0c0-7bf76dbabaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 12:01:28.817606 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:28.817512 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 12:01:29.511690 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:29.511643 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-thwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 12:01:29.512214 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:29.511715 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" podUID="13ee911d-d862-4efa-b0c0-7bf76dbabaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 12:01:30.290757 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:30.290728 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:01:30.511486 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:30.511452 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-thwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 12:01:30.511748 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:30.511721 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" podUID="13ee911d-d862-4efa-b0c0-7bf76dbabaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 12:01:31.510815 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:31.510781 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-thwsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 12:01:31.511129 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:31.511091 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" podUID="13ee911d-d862-4efa-b0c0-7bf76dbabaab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 12:01:32.515095 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:32.515061 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 12:01:32.516191 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:32.516170 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-thwsf" Sep 10 12:01:32.575317 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:32.575291 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 12:01:32.579958 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:32.579842 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 12:01:46.055485 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:46.055449 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:01:53.954179 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:53.954092 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd"] Sep 10 12:01:53.954672 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:53.954653 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd321a7-4513-4ae4-8db7-1b664871f133" containerName="authorino" Sep 10 12:01:53.954717 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:53.954677 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd321a7-4513-4ae4-8db7-1b664871f133" containerName="authorino" Sep 10 12:01:53.954827 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:53.954806 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd321a7-4513-4ae4-8db7-1b664871f133" containerName="authorino" Sep 10 12:01:53.957329 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:53.957311 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:53.961336 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:53.961313 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 12:01:53.961448 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:53.961370 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wfrs7\"" Sep 10 12:01:53.961520 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:53.961367 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 12:01:53.961520 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:53.961380 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 10 12:01:53.969518 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:53.969495 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd"] Sep 10 12:01:54.118693 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.118660 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvrhk\" (UniqueName: \"kubernetes.io/projected/898633e3-8f5b-4a41-98fc-81bdbd50bd21-kube-api-access-kvrhk\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.118867 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.118704 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/898633e3-8f5b-4a41-98fc-81bdbd50bd21-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.118867 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.118724 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898633e3-8f5b-4a41-98fc-81bdbd50bd21-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.118867 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.118791 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/898633e3-8f5b-4a41-98fc-81bdbd50bd21-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.118867 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.118813 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898633e3-8f5b-4a41-98fc-81bdbd50bd21-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.118867 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.118858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898633e3-8f5b-4a41-98fc-81bdbd50bd21-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.220124 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.220044 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvrhk\" (UniqueName: \"kubernetes.io/projected/898633e3-8f5b-4a41-98fc-81bdbd50bd21-kube-api-access-kvrhk\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.220124 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.220086 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/898633e3-8f5b-4a41-98fc-81bdbd50bd21-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.220124 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.220107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898633e3-8f5b-4a41-98fc-81bdbd50bd21-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.220389 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.220167 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/898633e3-8f5b-4a41-98fc-81bdbd50bd21-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.220389 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.220200 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898633e3-8f5b-4a41-98fc-81bdbd50bd21-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.220389 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.220246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898633e3-8f5b-4a41-98fc-81bdbd50bd21-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.220573 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.220554 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898633e3-8f5b-4a41-98fc-81bdbd50bd21-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.220617 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.220558 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/898633e3-8f5b-4a41-98fc-81bdbd50bd21-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.220654 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.220626 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898633e3-8f5b-4a41-98fc-81bdbd50bd21-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.222580 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.222560 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/898633e3-8f5b-4a41-98fc-81bdbd50bd21-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.222749 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.222732 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898633e3-8f5b-4a41-98fc-81bdbd50bd21-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.228355 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.228335 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvrhk\" (UniqueName: \"kubernetes.io/projected/898633e3-8f5b-4a41-98fc-81bdbd50bd21-kube-api-access-kvrhk\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd\" (UID: \"898633e3-8f5b-4a41-98fc-81bdbd50bd21\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.269408 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.269375 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:01:54.552379 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.552349 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd"] Sep 10 12:01:54.554724 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:01:54.554695 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898633e3_8f5b_4a41_98fc_81bdbd50bd21.slice/crio-16573137b6a38233dbeb86b8980c78bd7fcff93c2a09846c87e5b2a10a0d1dcf WatchSource:0}: Error finding container 16573137b6a38233dbeb86b8980c78bd7fcff93c2a09846c87e5b2a10a0d1dcf: Status 404 returned error can't find the container with id 16573137b6a38233dbeb86b8980c78bd7fcff93c2a09846c87e5b2a10a0d1dcf Sep 10 12:01:54.556356 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.556336 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 12:01:54.968162 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:54.968104 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:01:55.417471 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:55.417387 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" event={"ID":"898633e3-8f5b-4a41-98fc-81bdbd50bd21","Type":"ContainerStarted","Data":"16573137b6a38233dbeb86b8980c78bd7fcff93c2a09846c87e5b2a10a0d1dcf"} Sep 10 12:01:56.423702 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:56.423661 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" event={"ID":"898633e3-8f5b-4a41-98fc-81bdbd50bd21","Type":"ContainerStarted","Data":"43c4ce806de697e4c1775dabedb1798cc88741b2cac7b0011ae3f93032875e99"} Sep 10 12:01:56.449262 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:01:56.449214 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" podStartSLOduration=2.484055954 podStartE2EDuration="3.449196016s" totalImagesPullingTime="965.140062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:01:56.446849442 +0000 UTC m=+624.326238874" watchObservedRunningTime="2026-09-10 12:01:56.449196016 +0000 UTC m=+624.328585450" Sep 10 12:02:04.270304 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:04.270267 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:02:04.270804 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:04.270330 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:02:04.283080 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:04.283053 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:02:04.472805 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:04.472775 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd" Sep 10 12:02:23.892182 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:23.892137 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5"] Sep 10 12:02:23.894611 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:23.894594 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:23.899585 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:23.899551 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 10 12:02:23.913185 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:23.913161 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5"] Sep 10 12:02:23.979122 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:23.979089 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f541f120-87ff-4a37-a56f-365e603cf2d9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:23.979294 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:23.979123 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wvqs\" (UniqueName: \"kubernetes.io/projected/f541f120-87ff-4a37-a56f-365e603cf2d9-kube-api-access-7wvqs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:23.979294 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:23.979176 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f541f120-87ff-4a37-a56f-365e603cf2d9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:23.979294 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:23.979228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f541f120-87ff-4a37-a56f-365e603cf2d9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:23.979398 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:23.979295 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f541f120-87ff-4a37-a56f-365e603cf2d9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:23.979398 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:23.979318 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f541f120-87ff-4a37-a56f-365e603cf2d9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.080271 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.080237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f541f120-87ff-4a37-a56f-365e603cf2d9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.080474 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.080287 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f541f120-87ff-4a37-a56f-365e603cf2d9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.080474 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.080308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f541f120-87ff-4a37-a56f-365e603cf2d9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.080474 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.080381 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f541f120-87ff-4a37-a56f-365e603cf2d9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.080474 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.080403 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wvqs\" (UniqueName: \"kubernetes.io/projected/f541f120-87ff-4a37-a56f-365e603cf2d9-kube-api-access-7wvqs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.080474 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.080422 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f541f120-87ff-4a37-a56f-365e603cf2d9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.080740 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.080713 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f541f120-87ff-4a37-a56f-365e603cf2d9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.080740 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.080743 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f541f120-87ff-4a37-a56f-365e603cf2d9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.080838 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.080760 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f541f120-87ff-4a37-a56f-365e603cf2d9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.082869 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.082842 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f541f120-87ff-4a37-a56f-365e603cf2d9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.083198 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.083178 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f541f120-87ff-4a37-a56f-365e603cf2d9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.091008 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.090985 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wvqs\" (UniqueName: \"kubernetes.io/projected/f541f120-87ff-4a37-a56f-365e603cf2d9-kube-api-access-7wvqs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d57w5\" (UID: \"f541f120-87ff-4a37-a56f-365e603cf2d9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.204223 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.204191 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:24.483444 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.483415 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5"] Sep 10 12:02:24.485447 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:02:24.485423 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf541f120_87ff_4a37_a56f_365e603cf2d9.slice/crio-0275bd32b681c27a546e4b9139575c36afa9df40187700ff994ea2558619e08d WatchSource:0}: Error finding container 0275bd32b681c27a546e4b9139575c36afa9df40187700ff994ea2558619e08d: Status 404 returned error can't find the container with id 0275bd32b681c27a546e4b9139575c36afa9df40187700ff994ea2558619e08d Sep 10 12:02:24.563001 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:24.562964 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" event={"ID":"f541f120-87ff-4a37-a56f-365e603cf2d9","Type":"ContainerStarted","Data":"0275bd32b681c27a546e4b9139575c36afa9df40187700ff994ea2558619e08d"} Sep 10 12:02:25.187206 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:25.187169 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:02:25.570386 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:25.570295 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" event={"ID":"f541f120-87ff-4a37-a56f-365e603cf2d9","Type":"ContainerStarted","Data":"2b9a7005a01a981c46c68a546a8d0b41daca5137a05549b44a5a2ab89e0db76f"} Sep 10 12:02:25.590875 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:25.590815 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" podStartSLOduration=2.424729844 podStartE2EDuration="2.590798372s" totalImagesPullingTime="166.068528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:02:25.588484728 +0000 UTC m=+653.467874162" watchObservedRunningTime="2026-09-10 12:02:25.590798372 +0000 UTC m=+653.470187803" Sep 10 12:02:34.204636 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:34.204601 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:34.205047 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:34.204875 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:34.217258 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:34.217235 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:34.629674 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:34.629601 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d57w5" Sep 10 12:02:43.781584 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:43.781543 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:02:56.365994 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:02:56.365950 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:03:02.458740 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.458707 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8"] Sep 10 12:03:02.461217 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.461199 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.464099 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.464077 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 10 12:03:02.476864 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.476835 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8"] Sep 10 12:03:02.518995 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.518960 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.518995 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.518991 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.519204 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.519033 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.519204 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.519080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tznb9\" (UniqueName: \"kubernetes.io/projected/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-kube-api-access-tznb9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.519204 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.519114 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.519304 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.519207 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.620080 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.620043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.620291 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.620127 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.620291 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.620196 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.620291 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.620237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.620454 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.620297 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tznb9\" (UniqueName: \"kubernetes.io/projected/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-kube-api-access-tznb9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.620454 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.620407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.620640 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.620619 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.620764 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.620741 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.620815 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.620761 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.623100 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.623071 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.623363 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.623346 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.630959 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.630933 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tznb9\" (UniqueName: \"kubernetes.io/projected/35684a10-f5ad-4ef9-9c2c-af9a300a02d6-kube-api-access-tznb9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-68kk8\" (UID: \"35684a10-f5ad-4ef9-9c2c-af9a300a02d6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:02.771731 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:02.771647 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:03.053109 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:03.053085 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8"] Sep 10 12:03:03.054813 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:03:03.054789 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35684a10_f5ad_4ef9_9c2c_af9a300a02d6.slice/crio-591977330a861e4cb9026159380b24c7f41ecef6d7692d6a1f968c96e1fb9ece WatchSource:0}: Error finding container 591977330a861e4cb9026159380b24c7f41ecef6d7692d6a1f968c96e1fb9ece: Status 404 returned error can't find the container with id 591977330a861e4cb9026159380b24c7f41ecef6d7692d6a1f968c96e1fb9ece Sep 10 12:03:03.762909 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:03.762877 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" event={"ID":"35684a10-f5ad-4ef9-9c2c-af9a300a02d6","Type":"ContainerStarted","Data":"8f724a269ff85d2d0d0c1dda64267b37a3b6db3008651cb65fd562df208dcc6f"} Sep 10 12:03:03.763377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:03.762921 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" event={"ID":"35684a10-f5ad-4ef9-9c2c-af9a300a02d6","Type":"ContainerStarted","Data":"591977330a861e4cb9026159380b24c7f41ecef6d7692d6a1f968c96e1fb9ece"} Sep 10 12:03:03.783466 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:03.783414 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" podStartSLOduration=1.587501896 podStartE2EDuration="1.783399188s" totalImagesPullingTime="195.897292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:03:03.781975058 +0000 UTC m=+691.661364490" watchObservedRunningTime="2026-09-10 12:03:03.783399188 +0000 UTC m=+691.662788620" Sep 10 12:03:05.261976 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:05.261943 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:03:12.772495 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:12.772462 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:12.772495 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:12.772502 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:12.784796 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:12.784768 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:03:12.818522 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:03:12.818488 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-68kk8" Sep 10 12:04:16.204813 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:16.204778 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687c5c4ddc-lgd7p"] Sep 10 12:04:16.207041 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:16.207025 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687c5c4ddc-lgd7p" Sep 10 12:04:16.210758 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:16.210738 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-m6mmm\"" Sep 10 12:04:16.211957 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:16.211941 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 10 12:04:16.218601 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:16.218582 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687c5c4ddc-lgd7p"] Sep 10 12:04:16.280836 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:16.280807 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55ts\" (UniqueName: \"kubernetes.io/projected/e2244333-af65-4c1e-8ebb-8df9410490c1-kube-api-access-l55ts\") pod \"authorino-687c5c4ddc-lgd7p\" (UID: \"e2244333-af65-4c1e-8ebb-8df9410490c1\") " pod="kuadrant-system/authorino-687c5c4ddc-lgd7p" Sep 10 12:04:16.280982 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:16.280892 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2244333-af65-4c1e-8ebb-8df9410490c1-tls-cert\") pod \"authorino-687c5c4ddc-lgd7p\" (UID: \"e2244333-af65-4c1e-8ebb-8df9410490c1\") " pod="kuadrant-system/authorino-687c5c4ddc-lgd7p" Sep 10 12:04:16.382153 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:16.382111 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2244333-af65-4c1e-8ebb-8df9410490c1-tls-cert\") pod \"authorino-687c5c4ddc-lgd7p\" (UID: \"e2244333-af65-4c1e-8ebb-8df9410490c1\") " pod="kuadrant-system/authorino-687c5c4ddc-lgd7p" Sep 10 12:04:16.382320 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:16.382201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55ts\" (UniqueName: \"kubernetes.io/projected/e2244333-af65-4c1e-8ebb-8df9410490c1-kube-api-access-l55ts\") pod \"authorino-687c5c4ddc-lgd7p\" (UID: \"e2244333-af65-4c1e-8ebb-8df9410490c1\") " pod="kuadrant-system/authorino-687c5c4ddc-lgd7p" Sep 10 12:04:16.384641 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:16.384623 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2244333-af65-4c1e-8ebb-8df9410490c1-tls-cert\") pod \"authorino-687c5c4ddc-lgd7p\" (UID: \"e2244333-af65-4c1e-8ebb-8df9410490c1\") " pod="kuadrant-system/authorino-687c5c4ddc-lgd7p" Sep 10 12:04:16.391925 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:16.391903 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55ts\" (UniqueName: \"kubernetes.io/projected/e2244333-af65-4c1e-8ebb-8df9410490c1-kube-api-access-l55ts\") pod \"authorino-687c5c4ddc-lgd7p\" (UID: \"e2244333-af65-4c1e-8ebb-8df9410490c1\") " pod="kuadrant-system/authorino-687c5c4ddc-lgd7p" Sep 10 12:04:16.516564 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:16.516480 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687c5c4ddc-lgd7p" Sep 10 12:04:16.784354 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:16.784296 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687c5c4ddc-lgd7p"] Sep 10 12:04:16.786747 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:04:16.786721 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2244333_af65_4c1e_8ebb_8df9410490c1.slice/crio-38975fbc36435435e4b290e81a4c5683d1e4018b83b742cef8c14b0995f53587 WatchSource:0}: Error finding container 38975fbc36435435e4b290e81a4c5683d1e4018b83b742cef8c14b0995f53587: Status 404 returned error can't find the container with id 38975fbc36435435e4b290e81a4c5683d1e4018b83b742cef8c14b0995f53587 Sep 10 12:04:17.138840 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:17.138758 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687c5c4ddc-lgd7p" event={"ID":"e2244333-af65-4c1e-8ebb-8df9410490c1","Type":"ContainerStarted","Data":"38975fbc36435435e4b290e81a4c5683d1e4018b83b742cef8c14b0995f53587"} Sep 10 12:04:18.146828 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:18.146782 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687c5c4ddc-lgd7p" event={"ID":"e2244333-af65-4c1e-8ebb-8df9410490c1","Type":"ContainerStarted","Data":"0b76c57e8a225f81ae7cc279ebea8c3ac7f8c0243d64e63d3521889f4a8d64e2"} Sep 10 12:04:18.167667 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:18.167612 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-687c5c4ddc-lgd7p" podStartSLOduration=1.280785915 podStartE2EDuration="2.167596628s" totalImagesPullingTime="886.810713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:04:18.166266204 +0000 UTC m=+766.045655636" watchObservedRunningTime="2026-09-10 12:04:18.167596628 +0000 UTC m=+766.046986094" Sep 10 12:04:40.177806 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:40.177769 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:04:42.903898 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:42.903858 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7967-139929861815728"] Sep 10 12:04:42.906174 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:42.906132 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7967-139929861815728" Sep 10 12:04:42.908939 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:42.908918 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7cc6f\"" Sep 10 12:04:42.915460 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:42.915438 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7967-139929861815728"] Sep 10 12:04:42.932649 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:42.932623 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgl6g\" (UniqueName: \"kubernetes.io/projected/3a01d58c-ba3b-497e-a0d8-9155e9bb2d57-kube-api-access-hgl6g\") pod \"test-curl-7967-139929861815728\" (UID: \"3a01d58c-ba3b-497e-a0d8-9155e9bb2d57\") " pod="opendatahub/test-curl-7967-139929861815728" Sep 10 12:04:43.034226 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:43.034193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgl6g\" (UniqueName: \"kubernetes.io/projected/3a01d58c-ba3b-497e-a0d8-9155e9bb2d57-kube-api-access-hgl6g\") pod \"test-curl-7967-139929861815728\" (UID: \"3a01d58c-ba3b-497e-a0d8-9155e9bb2d57\") " pod="opendatahub/test-curl-7967-139929861815728" Sep 10 12:04:43.042682 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:43.042658 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgl6g\" (UniqueName: \"kubernetes.io/projected/3a01d58c-ba3b-497e-a0d8-9155e9bb2d57-kube-api-access-hgl6g\") pod \"test-curl-7967-139929861815728\" (UID: \"3a01d58c-ba3b-497e-a0d8-9155e9bb2d57\") " pod="opendatahub/test-curl-7967-139929861815728" Sep 10 12:04:43.216333 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:43.216301 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7967-139929861815728" Sep 10 12:04:43.485801 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:43.485775 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7967-139929861815728"] Sep 10 12:04:43.488273 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:04:43.488247 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a01d58c_ba3b_497e_a0d8_9155e9bb2d57.slice/crio-2d33ef97d7eea5505f2f7216596d53624c64a16ae24fbb342ab71a1ec510a8a7 WatchSource:0}: Error finding container 2d33ef97d7eea5505f2f7216596d53624c64a16ae24fbb342ab71a1ec510a8a7: Status 404 returned error can't find the container with id 2d33ef97d7eea5505f2f7216596d53624c64a16ae24fbb342ab71a1ec510a8a7 Sep 10 12:04:44.282202 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.282167 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7967-139929861815728" event={"ID":"3a01d58c-ba3b-497e-a0d8-9155e9bb2d57","Type":"ContainerStarted","Data":"2d33ef97d7eea5505f2f7216596d53624c64a16ae24fbb342ab71a1ec510a8a7"} Sep 10 12:04:44.595956 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.595595 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj"] Sep 10 12:04:44.598474 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.598453 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.601960 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.601938 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7a26e68a-openshift-default-dockercfg-phc8c\"" Sep 10 12:04:44.618208 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.618179 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj"] Sep 10 12:04:44.650088 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.650048 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-data\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.650088 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.650087 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-envoy\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.650297 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.650121 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-workload-socket\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.650297 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.650233 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cc736f-12d1-4505-8611-5ecda7405a2f-istiod-ca-cert\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.650297 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.650285 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znwkx\" (UniqueName: \"kubernetes.io/projected/50cc736f-12d1-4505-8611-5ecda7405a2f-kube-api-access-znwkx\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.650420 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.650333 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-workload-certs\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.650420 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.650367 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-podinfo\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.650420 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.650398 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-credential-socket\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.650542 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.650422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-token\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.751367 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.751329 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-data\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.751576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.751379 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-envoy\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.751576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.751415 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-workload-socket\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.751576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.751457 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cc736f-12d1-4505-8611-5ecda7405a2f-istiod-ca-cert\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.751576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.751498 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znwkx\" (UniqueName: \"kubernetes.io/projected/50cc736f-12d1-4505-8611-5ecda7405a2f-kube-api-access-znwkx\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.751576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.751526 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-workload-certs\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.751576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.751556 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-podinfo\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.751890 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.751584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-credential-socket\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.751890 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.751605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-token\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.751890 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.751802 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-data\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.751890 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.751865 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-workload-socket\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.752084 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.752057 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-workload-certs\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.752134 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.752095 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-credential-socket\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.752347 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.752326 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cc736f-12d1-4505-8611-5ecda7405a2f-istiod-ca-cert\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.754247 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.754227 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-envoy\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.754589 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.754564 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-podinfo\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.761447 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.761423 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-token\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.762394 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.762373 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znwkx\" (UniqueName: \"kubernetes.io/projected/50cc736f-12d1-4505-8611-5ecda7405a2f-kube-api-access-znwkx\") pod \"e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:44.911478 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:44.911390 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:45.202428 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:45.202388 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj"] Sep 10 12:04:45.666345 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:04:45.666271 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cc736f_12d1_4505_8611_5ecda7405a2f.slice/crio-c7620afbbfa0f57a174c418f94d261af520e89e8c5b53190e3cf73e2cd0c3467 WatchSource:0}: Error finding container c7620afbbfa0f57a174c418f94d261af520e89e8c5b53190e3cf73e2cd0c3467: Status 404 returned error can't find the container with id c7620afbbfa0f57a174c418f94d261af520e89e8c5b53190e3cf73e2cd0c3467 Sep 10 12:04:45.668858 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:45.668818 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:04:45.668956 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:45.668908 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:04:45.669017 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:45.668955 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:04:46.296105 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:46.296073 2582 generic.go:356] "Generic (PLEG): container finished" podID="3a01d58c-ba3b-497e-a0d8-9155e9bb2d57" containerID="6c0fa26f7ef98506ae47971e9e00a4cf765c35ebb6d7a346b0e34bd1d361c36a" exitCode=0 Sep 10 12:04:46.296330 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:46.296184 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7967-139929861815728" event={"ID":"3a01d58c-ba3b-497e-a0d8-9155e9bb2d57","Type":"ContainerDied","Data":"6c0fa26f7ef98506ae47971e9e00a4cf765c35ebb6d7a346b0e34bd1d361c36a"} Sep 10 12:04:46.298189 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:46.298130 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" event={"ID":"50cc736f-12d1-4505-8611-5ecda7405a2f","Type":"ContainerStarted","Data":"ca8ffc54a67427ca6ce660375c7eb7e0a20db5e1fbff4154be815db5c27f7353"} Sep 10 12:04:46.298331 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:46.298215 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" event={"ID":"50cc736f-12d1-4505-8611-5ecda7405a2f","Type":"ContainerStarted","Data":"c7620afbbfa0f57a174c418f94d261af520e89e8c5b53190e3cf73e2cd0c3467"} Sep 10 12:04:46.335012 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:46.334966 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" podStartSLOduration=2.334952863 podStartE2EDuration="2.334952863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:04:46.331844379 +0000 UTC m=+794.211233811" watchObservedRunningTime="2026-09-10 12:04:46.334952863 +0000 UTC m=+794.214342295" Sep 10 12:04:46.912164 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:46.912108 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:46.917310 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:46.917282 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:47.303952 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:47.303926 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:47.305066 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:47.305045 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:04:47.488062 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:47.488037 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7967-139929861815728" Sep 10 12:04:47.556417 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:47.556328 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7967-139929861815728_3a01d58c-ba3b-497e-a0d8-9155e9bb2d57/test-curl-7967-139929861815728/0.log" Sep 10 12:04:47.580347 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:47.580318 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7967-139929861815728"] Sep 10 12:04:47.581506 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:47.581481 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgl6g\" (UniqueName: \"kubernetes.io/projected/3a01d58c-ba3b-497e-a0d8-9155e9bb2d57-kube-api-access-hgl6g\") pod \"3a01d58c-ba3b-497e-a0d8-9155e9bb2d57\" (UID: \"3a01d58c-ba3b-497e-a0d8-9155e9bb2d57\") " Sep 10 12:04:47.583721 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:47.583698 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a01d58c-ba3b-497e-a0d8-9155e9bb2d57-kube-api-access-hgl6g" (OuterVolumeSpecName: "kube-api-access-hgl6g") pod "3a01d58c-ba3b-497e-a0d8-9155e9bb2d57" (UID: "3a01d58c-ba3b-497e-a0d8-9155e9bb2d57"). InnerVolumeSpecName "kube-api-access-hgl6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:04:47.590201 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:47.590176 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7967-139929861815728"] Sep 10 12:04:47.683060 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:47.683028 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgl6g\" (UniqueName: \"kubernetes.io/projected/3a01d58c-ba3b-497e-a0d8-9155e9bb2d57-kube-api-access-hgl6g\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:04:48.311294 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:48.311267 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d33ef97d7eea5505f2f7216596d53624c64a16ae24fbb342ab71a1ec510a8a7" Sep 10 12:04:48.311294 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:48.311293 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7967-139929861815728" Sep 10 12:04:48.321996 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:04:48.321965 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7967-139929861815728\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="3a01d58c-ba3b-497e-a0d8-9155e9bb2d57" pod="opendatahub/test-curl-7967-139929861815728" Sep 10 12:04:48.633043 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:48.632956 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a01d58c-ba3b-497e-a0d8-9155e9bb2d57" path="/var/lib/kubelet/pods/3a01d58c-ba3b-497e-a0d8-9155e9bb2d57/volumes" Sep 10 12:04:53.871406 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:04:53.871326 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:05:03.345408 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:03.345341 2582 patch_prober.go:28] interesting pod/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 12:05:03.345917 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:03.345417 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 12:05:04.404221 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:04.404115 2582 generic.go:356] "Generic (PLEG): container finished" podID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerID="ca8ffc54a67427ca6ce660375c7eb7e0a20db5e1fbff4154be815db5c27f7353" exitCode=0 Sep 10 12:05:04.404221 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:04.404172 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" event={"ID":"50cc736f-12d1-4505-8611-5ecda7405a2f","Type":"ContainerDied","Data":"ca8ffc54a67427ca6ce660375c7eb7e0a20db5e1fbff4154be815db5c27f7353"} Sep 10 12:05:04.404673 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:04.404592 2582 scope.go:117] "RemoveContainer" containerID="ca8ffc54a67427ca6ce660375c7eb7e0a20db5e1fbff4154be815db5c27f7353" Sep 10 12:05:04.405769 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:04.405741 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:04.405855 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:04.405840 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:04.405897 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:04.405880 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:05.133427 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.133394 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc"] Sep 10 12:05:05.133935 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.133919 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a01d58c-ba3b-497e-a0d8-9155e9bb2d57" containerName="test-curl-7967-139929861815728" Sep 10 12:05:05.133997 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.133936 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a01d58c-ba3b-497e-a0d8-9155e9bb2d57" containerName="test-curl-7967-139929861815728" Sep 10 12:05:05.134036 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.134009 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a01d58c-ba3b-497e-a0d8-9155e9bb2d57" containerName="test-curl-7967-139929861815728" Sep 10 12:05:05.137069 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.137054 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.139772 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.139751 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-425dc7-openshift-default-dockercfg-m2qgr\"" Sep 10 12:05:05.150306 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.150285 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc"] Sep 10 12:05:05.255670 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.255617 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-credential-socket\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.255862 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.255678 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-data\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.255862 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.255729 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3c479c-8910-473c-a499-8b9e8d8d10b6-istiod-ca-cert\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.255862 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.255762 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-workload-certs\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.255862 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.255786 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-token\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.255862 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.255810 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrttd\" (UniqueName: \"kubernetes.io/projected/db3c479c-8910-473c-a499-8b9e8d8d10b6-kube-api-access-hrttd\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.255862 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.255834 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-envoy\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.256185 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.255979 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-podinfo\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.256185 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.256030 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-workload-socket\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.356805 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.356771 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-credential-socket\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.356805 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.356807 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-data\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.357004 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.356825 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3c479c-8910-473c-a499-8b9e8d8d10b6-istiod-ca-cert\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.357103 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.357083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-workload-certs\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.357188 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.357128 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-token\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.357188 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.357174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrttd\" (UniqueName: \"kubernetes.io/projected/db3c479c-8910-473c-a499-8b9e8d8d10b6-kube-api-access-hrttd\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.357308 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.357195 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-data\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.357308 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.357203 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-envoy\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.357308 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.357265 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-credential-socket\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.357468 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.357363 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-podinfo\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.357468 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.357400 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-workload-certs\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.357468 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.357419 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-workload-socket\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.357660 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.357644 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3c479c-8910-473c-a499-8b9e8d8d10b6-istiod-ca-cert\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.357735 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.357714 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-workload-socket\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.359458 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.359436 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-envoy\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.359713 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.359696 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-podinfo\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.365750 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.365727 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-token\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.365946 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.365926 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrttd\" (UniqueName: \"kubernetes.io/projected/db3c479c-8910-473c-a499-8b9e8d8d10b6-kube-api-access-hrttd\") pod \"e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.410909 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.410818 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" event={"ID":"50cc736f-12d1-4505-8611-5ecda7405a2f","Type":"ContainerStarted","Data":"09258415ec28fef6ec9dc59a6fd21ef94815b1164753abcd00e0a4074d896e58"} Sep 10 12:05:05.447772 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.447745 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:05.752827 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.752801 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc"] Sep 10 12:05:05.754488 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:05:05.754456 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3c479c_8910_473c_a499_8b9e8d8d10b6.slice/crio-59097d919e45b9bb674e2f78286c0a6fb532cdfe2f3e48a7a1ee14d1eb066320 WatchSource:0}: Error finding container 59097d919e45b9bb674e2f78286c0a6fb532cdfe2f3e48a7a1ee14d1eb066320: Status 404 returned error can't find the container with id 59097d919e45b9bb674e2f78286c0a6fb532cdfe2f3e48a7a1ee14d1eb066320 Sep 10 12:05:05.756874 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.756841 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:05.756976 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.756915 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:05.756976 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.756945 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:05.912010 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.911965 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:05:05.912407 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.912376 2582 patch_prober.go:28] interesting pod/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 12:05:05.912512 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:05.912447 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 12:05:06.421858 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:06.421812 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" event={"ID":"db3c479c-8910-473c-a499-8b9e8d8d10b6","Type":"ContainerStarted","Data":"8b0eb934823c024958fcdd1566d4473a95aca55a0331e01f3aba07db61018e82"} Sep 10 12:05:06.422386 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:06.421884 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" event={"ID":"db3c479c-8910-473c-a499-8b9e8d8d10b6","Type":"ContainerStarted","Data":"59097d919e45b9bb674e2f78286c0a6fb532cdfe2f3e48a7a1ee14d1eb066320"} Sep 10 12:05:06.448170 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:06.448118 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:06.448430 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:06.448393 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" podStartSLOduration=1.448378892 podStartE2EDuration="1.448378892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:05:06.445848257 +0000 UTC m=+814.325237689" watchObservedRunningTime="2026-09-10 12:05:06.448378892 +0000 UTC m=+814.327768324" Sep 10 12:05:06.452977 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:06.452955 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:06.912107 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:06.912060 2582 patch_prober.go:28] interesting pod/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 12:05:06.912337 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:06.912161 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 12:05:07.429119 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:07.429091 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:07.430048 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:07.430028 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:07.911870 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:07.911833 2582 patch_prober.go:28] interesting pod/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 12:05:07.912081 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:07.911902 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 12:05:08.912231 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:08.912192 2582 patch_prober.go:28] interesting pod/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 12:05:08.912822 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:08.912790 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 12:05:09.912128 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:09.912097 2582 patch_prober.go:28] interesting pod/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 12:05:09.912426 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:09.912399 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 12:05:10.912743 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:10.912641 2582 patch_prober.go:28] interesting pod/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 12:05:10.912743 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:10.912708 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 12:05:11.916043 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:11.916011 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:05:11.916477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:11.916391 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:05:11.917235 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:11.917217 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:05:13.665097 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:13.665061 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:05:18.449511 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:18.449474 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb"] Sep 10 12:05:18.456270 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:18.456249 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb" Sep 10 12:05:18.459105 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:18.459080 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:05:18.459241 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:18.459087 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:05:18.462993 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:18.462967 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb"] Sep 10 12:05:18.515170 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:18.514685 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f434ad-be74-496d-a919-ddef8235517e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb\" (UID: \"53f434ad-be74-496d-a919-ddef8235517e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb" Sep 10 12:05:18.615400 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:18.615360 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f434ad-be74-496d-a919-ddef8235517e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb\" (UID: \"53f434ad-be74-496d-a919-ddef8235517e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb" Sep 10 12:05:18.616250 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:18.616216 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f434ad-be74-496d-a919-ddef8235517e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb\" (UID: \"53f434ad-be74-496d-a919-ddef8235517e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb" Sep 10 12:05:18.768925 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:18.768838 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb" Sep 10 12:05:19.067101 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:19.067029 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb"] Sep 10 12:05:19.070687 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:05:19.070647 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f434ad_be74_496d_a919_ddef8235517e.slice/crio-4cbdfd569be870a48ffd020ced4532511b99acc26b110243919dfd731c8ed357 WatchSource:0}: Error finding container 4cbdfd569be870a48ffd020ced4532511b99acc26b110243919dfd731c8ed357: Status 404 returned error can't find the container with id 4cbdfd569be870a48ffd020ced4532511b99acc26b110243919dfd731c8ed357 Sep 10 12:05:19.501095 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:19.501047 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb" event={"ID":"53f434ad-be74-496d-a919-ddef8235517e","Type":"ContainerStarted","Data":"4cbdfd569be870a48ffd020ced4532511b99acc26b110243919dfd731c8ed357"} Sep 10 12:05:23.470350 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:23.470300 2582 patch_prober.go:28] interesting pod/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 12:05:23.470745 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:23.470370 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 12:05:23.545155 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:23.545108 2582 generic.go:356] "Generic (PLEG): container finished" podID="53f434ad-be74-496d-a919-ddef8235517e" containerID="b1e0ad307bd4f9d6c4fca92a852f6505dedd3b737ce0d827e2eebab373316054" exitCode=0 Sep 10 12:05:23.545312 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:23.545180 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb" event={"ID":"53f434ad-be74-496d-a919-ddef8235517e","Type":"ContainerDied","Data":"b1e0ad307bd4f9d6c4fca92a852f6505dedd3b737ce0d827e2eebab373316054"} Sep 10 12:05:24.552824 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:24.552718 2582 generic.go:356] "Generic (PLEG): container finished" podID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerID="8b0eb934823c024958fcdd1566d4473a95aca55a0331e01f3aba07db61018e82" exitCode=0 Sep 10 12:05:24.552824 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:24.552800 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" event={"ID":"db3c479c-8910-473c-a499-8b9e8d8d10b6","Type":"ContainerDied","Data":"8b0eb934823c024958fcdd1566d4473a95aca55a0331e01f3aba07db61018e82"} Sep 10 12:05:24.553862 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:24.553401 2582 scope.go:117] "RemoveContainer" containerID="8b0eb934823c024958fcdd1566d4473a95aca55a0331e01f3aba07db61018e82" Sep 10 12:05:24.554833 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:24.554793 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:24.554923 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:24.554882 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:24.554923 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:24.554914 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:24.730617 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:24.730592 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb" Sep 10 12:05:24.781274 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:24.779796 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f434ad-be74-496d-a919-ddef8235517e-maas-api-service-ca\") pod \"53f434ad-be74-496d-a919-ddef8235517e\" (UID: \"53f434ad-be74-496d-a919-ddef8235517e\") " Sep 10 12:05:24.781274 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:24.780269 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f434ad-be74-496d-a919-ddef8235517e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53f434ad-be74-496d-a919-ddef8235517e" (UID: "53f434ad-be74-496d-a919-ddef8235517e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:05:24.881592 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:24.881501 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f434ad-be74-496d-a919-ddef8235517e-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:05:25.559631 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:25.559597 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" event={"ID":"db3c479c-8910-473c-a499-8b9e8d8d10b6","Type":"ContainerStarted","Data":"1f6219b7814fed0acb2daddd08c8d3defa7b79d8c7366c7739ad0706c6f9e67a"} Sep 10 12:05:25.560870 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:25.560848 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb" event={"ID":"53f434ad-be74-496d-a919-ddef8235517e","Type":"ContainerDied","Data":"4cbdfd569be870a48ffd020ced4532511b99acc26b110243919dfd731c8ed357"} Sep 10 12:05:25.560960 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:25.560880 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb" Sep 10 12:05:25.560960 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:25.560891 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cbdfd569be870a48ffd020ced4532511b99acc26b110243919dfd731c8ed357" Sep 10 12:05:26.448584 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:26.448531 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:26.448796 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:26.448725 2582 patch_prober.go:28] interesting pod/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 10 12:05:26.448796 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:26.448778 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 10 12:05:27.448557 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:27.448512 2582 patch_prober.go:28] interesting pod/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 10 12:05:27.448979 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:27.448582 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 10 12:05:28.449133 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:28.449092 2582 patch_prober.go:28] interesting pod/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 10 12:05:28.449557 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:28.449173 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 10 12:05:29.448362 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:29.448328 2582 patch_prober.go:28] interesting pod/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 10 12:05:29.448534 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:29.448382 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 10 12:05:29.786119 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:29.786045 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb"] Sep 10 12:05:29.789161 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:29.789119 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a26e68a-l7blb"] Sep 10 12:05:29.935931 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:29.935881 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj"] Sep 10 12:05:29.938189 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:29.937474 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" containerID="cri-o://09258415ec28fef6ec9dc59a6fd21ef94815b1164753abcd00e0a4074d896e58" gracePeriod=30 Sep 10 12:05:30.448553 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:30.448522 2582 patch_prober.go:28] interesting pod/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 10 12:05:30.448785 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:30.448764 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 10 12:05:30.630440 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:30.630403 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53f434ad-be74-496d-a919-ddef8235517e" path="/var/lib/kubelet/pods/53f434ad-be74-496d-a919-ddef8235517e/volumes" Sep 10 12:05:30.954858 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:30.954816 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw"] Sep 10 12:05:30.955463 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:30.955446 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53f434ad-be74-496d-a919-ddef8235517e" containerName="revoke-keys" Sep 10 12:05:30.955513 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:30.955466 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="53f434ad-be74-496d-a919-ddef8235517e" containerName="revoke-keys" Sep 10 12:05:30.955589 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:30.955579 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="53f434ad-be74-496d-a919-ddef8235517e" containerName="revoke-keys" Sep 10 12:05:30.959375 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:30.959350 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:30.963326 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:30.963300 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-17efa977-gw-openshift-default-dockercfg-fb6cs\"" Sep 10 12:05:30.976709 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:30.976682 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw"] Sep 10 12:05:31.047442 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.047407 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-workload-socket\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.047442 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.047446 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-podinfo\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.047690 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.047471 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-envoy\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.047690 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.047492 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csbl2\" (UniqueName: \"kubernetes.io/projected/6a8e1df5-54f9-4321-803a-6b40a3efda28-kube-api-access-csbl2\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.047690 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.047598 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-token\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.047690 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.047644 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-data\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.047690 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.047663 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e1df5-54f9-4321-803a-6b40a3efda28-istiod-ca-cert\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.047966 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.047746 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-credential-socket\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.047966 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.047795 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-workload-certs\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.148667 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.148626 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-envoy\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.148667 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.148669 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csbl2\" (UniqueName: \"kubernetes.io/projected/6a8e1df5-54f9-4321-803a-6b40a3efda28-kube-api-access-csbl2\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.148922 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.148704 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-token\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.148922 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.148750 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-data\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.148922 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.148769 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e1df5-54f9-4321-803a-6b40a3efda28-istiod-ca-cert\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.148922 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.148793 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-credential-socket\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.148922 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.148824 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-workload-certs\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.149217 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.148928 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-workload-socket\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.149217 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.148965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-podinfo\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.149311 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.149227 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-data\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.149367 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.149307 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-credential-socket\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.149367 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.149333 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-workload-certs\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.149439 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.149420 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-workload-socket\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.149582 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.149562 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e1df5-54f9-4321-803a-6b40a3efda28-istiod-ca-cert\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.151211 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.151183 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-envoy\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.151478 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.151455 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-podinfo\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.163471 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.163450 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csbl2\" (UniqueName: \"kubernetes.io/projected/6a8e1df5-54f9-4321-803a-6b40a3efda28-kube-api-access-csbl2\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.168321 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.168302 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-token\") pod \"e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.276504 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.276413 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:31.452054 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.452023 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:31.452280 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.452259 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:31.453069 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.453051 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:05:31.556343 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.556320 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw"] Sep 10 12:05:31.559665 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:05:31.559631 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8e1df5_54f9_4321_803a_6b40a3efda28.slice/crio-ee12776e54c99c690ae635b9025a13d27c815fab3a2835d659b785071013065e WatchSource:0}: Error finding container ee12776e54c99c690ae635b9025a13d27c815fab3a2835d659b785071013065e: Status 404 returned error can't find the container with id ee12776e54c99c690ae635b9025a13d27c815fab3a2835d659b785071013065e Sep 10 12:05:31.561833 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.561790 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:31.561915 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.561864 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:31.561915 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.561893 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:31.604255 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:31.604172 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" event={"ID":"6a8e1df5-54f9-4321-803a-6b40a3efda28","Type":"ContainerStarted","Data":"ee12776e54c99c690ae635b9025a13d27c815fab3a2835d659b785071013065e"} Sep 10 12:05:32.611040 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:32.611003 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" event={"ID":"6a8e1df5-54f9-4321-803a-6b40a3efda28","Type":"ContainerStarted","Data":"1ae34138c88f378b1f25e5fe8a86eec474c37343c7b6c0ed86b15408bfaa5c07"} Sep 10 12:05:32.632797 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:32.632739 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" podStartSLOduration=2.6327199869999998 podStartE2EDuration="2.632719987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:05:32.631608245 +0000 UTC m=+840.510997679" watchObservedRunningTime="2026-09-10 12:05:32.632719987 +0000 UTC m=+840.512109420" Sep 10 12:05:33.276685 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:33.276645 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:33.281898 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:33.281869 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:33.616829 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:33.616748 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:33.617694 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:33.617673 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:33.752075 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:33.752035 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:05:35.225041 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.225010 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:05:35.291889 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.291861 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-workload-certs\") pod \"50cc736f-12d1-4505-8611-5ecda7405a2f\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " Sep 10 12:05:35.292045 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.291950 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cc736f-12d1-4505-8611-5ecda7405a2f-istiod-ca-cert\") pod \"50cc736f-12d1-4505-8611-5ecda7405a2f\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " Sep 10 12:05:35.292045 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.291972 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-data\") pod \"50cc736f-12d1-4505-8611-5ecda7405a2f\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " Sep 10 12:05:35.292045 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.292000 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-podinfo\") pod \"50cc736f-12d1-4505-8611-5ecda7405a2f\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " Sep 10 12:05:35.292045 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.292015 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-envoy\") pod \"50cc736f-12d1-4505-8611-5ecda7405a2f\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " Sep 10 12:05:35.292355 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.292046 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-token\") pod \"50cc736f-12d1-4505-8611-5ecda7405a2f\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " Sep 10 12:05:35.292355 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.292081 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-credential-socket\") pod \"50cc736f-12d1-4505-8611-5ecda7405a2f\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " Sep 10 12:05:35.292355 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.292130 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znwkx\" (UniqueName: \"kubernetes.io/projected/50cc736f-12d1-4505-8611-5ecda7405a2f-kube-api-access-znwkx\") pod \"50cc736f-12d1-4505-8611-5ecda7405a2f\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " Sep 10 12:05:35.292355 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.292125 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "50cc736f-12d1-4505-8611-5ecda7405a2f" (UID: "50cc736f-12d1-4505-8611-5ecda7405a2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:05:35.292355 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.292194 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-workload-socket\") pod \"50cc736f-12d1-4505-8611-5ecda7405a2f\" (UID: \"50cc736f-12d1-4505-8611-5ecda7405a2f\") " Sep 10 12:05:35.292355 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.292326 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cc736f-12d1-4505-8611-5ecda7405a2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "50cc736f-12d1-4505-8611-5ecda7405a2f" (UID: "50cc736f-12d1-4505-8611-5ecda7405a2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:05:35.292673 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.292476 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "50cc736f-12d1-4505-8611-5ecda7405a2f" (UID: "50cc736f-12d1-4505-8611-5ecda7405a2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:05:35.292673 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.292502 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "50cc736f-12d1-4505-8611-5ecda7405a2f" (UID: "50cc736f-12d1-4505-8611-5ecda7405a2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:05:35.292673 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.292544 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "50cc736f-12d1-4505-8611-5ecda7405a2f" (UID: "50cc736f-12d1-4505-8611-5ecda7405a2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:05:35.292673 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.292549 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:05:35.292673 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.292567 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cc736f-12d1-4505-8611-5ecda7405a2f-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:05:35.300889 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.298279 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cc736f-12d1-4505-8611-5ecda7405a2f-kube-api-access-znwkx" (OuterVolumeSpecName: "kube-api-access-znwkx") pod "50cc736f-12d1-4505-8611-5ecda7405a2f" (UID: "50cc736f-12d1-4505-8611-5ecda7405a2f"). InnerVolumeSpecName "kube-api-access-znwkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:05:35.300889 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.300583 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "50cc736f-12d1-4505-8611-5ecda7405a2f" (UID: "50cc736f-12d1-4505-8611-5ecda7405a2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:05:35.300889 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.300590 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "50cc736f-12d1-4505-8611-5ecda7405a2f" (UID: "50cc736f-12d1-4505-8611-5ecda7405a2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:05:35.304271 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.304238 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "50cc736f-12d1-4505-8611-5ecda7405a2f" (UID: "50cc736f-12d1-4505-8611-5ecda7405a2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:05:35.393953 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.393921 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:05:35.393953 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.393946 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:05:35.393953 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.393955 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:05:35.393953 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.393962 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cc736f-12d1-4505-8611-5ecda7405a2f-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:05:35.394276 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.393971 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:05:35.394276 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.393982 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znwkx\" (UniqueName: \"kubernetes.io/projected/50cc736f-12d1-4505-8611-5ecda7405a2f-kube-api-access-znwkx\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:05:35.394276 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.393990 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cc736f-12d1-4505-8611-5ecda7405a2f-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:05:35.631708 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.631604 2582 generic.go:356] "Generic (PLEG): container finished" podID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerID="09258415ec28fef6ec9dc59a6fd21ef94815b1164753abcd00e0a4074d896e58" exitCode=0 Sep 10 12:05:35.631708 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.631665 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" Sep 10 12:05:35.631708 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.631696 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" event={"ID":"50cc736f-12d1-4505-8611-5ecda7405a2f","Type":"ContainerDied","Data":"09258415ec28fef6ec9dc59a6fd21ef94815b1164753abcd00e0a4074d896e58"} Sep 10 12:05:35.631992 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.631742 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj" event={"ID":"50cc736f-12d1-4505-8611-5ecda7405a2f","Type":"ContainerDied","Data":"c7620afbbfa0f57a174c418f94d261af520e89e8c5b53190e3cf73e2cd0c3467"} Sep 10 12:05:35.631992 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.631756 2582 scope.go:117] "RemoveContainer" containerID="09258415ec28fef6ec9dc59a6fd21ef94815b1164753abcd00e0a4074d896e58" Sep 10 12:05:35.641768 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.641738 2582 scope.go:117] "RemoveContainer" containerID="ca8ffc54a67427ca6ce660375c7eb7e0a20db5e1fbff4154be815db5c27f7353" Sep 10 12:05:35.650726 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.650707 2582 scope.go:117] "RemoveContainer" containerID="09258415ec28fef6ec9dc59a6fd21ef94815b1164753abcd00e0a4074d896e58" Sep 10 12:05:35.651040 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:05:35.651020 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09258415ec28fef6ec9dc59a6fd21ef94815b1164753abcd00e0a4074d896e58\": container with ID starting with 09258415ec28fef6ec9dc59a6fd21ef94815b1164753abcd00e0a4074d896e58 not found: ID does not exist" containerID="09258415ec28fef6ec9dc59a6fd21ef94815b1164753abcd00e0a4074d896e58" Sep 10 12:05:35.651099 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.651051 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09258415ec28fef6ec9dc59a6fd21ef94815b1164753abcd00e0a4074d896e58"} err="failed to get container status \"09258415ec28fef6ec9dc59a6fd21ef94815b1164753abcd00e0a4074d896e58\": rpc error: code = NotFound desc = could not find container \"09258415ec28fef6ec9dc59a6fd21ef94815b1164753abcd00e0a4074d896e58\": container with ID starting with 09258415ec28fef6ec9dc59a6fd21ef94815b1164753abcd00e0a4074d896e58 not found: ID does not exist" Sep 10 12:05:35.651099 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.651070 2582 scope.go:117] "RemoveContainer" containerID="ca8ffc54a67427ca6ce660375c7eb7e0a20db5e1fbff4154be815db5c27f7353" Sep 10 12:05:35.651376 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:05:35.651358 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8ffc54a67427ca6ce660375c7eb7e0a20db5e1fbff4154be815db5c27f7353\": container with ID starting with ca8ffc54a67427ca6ce660375c7eb7e0a20db5e1fbff4154be815db5c27f7353 not found: ID does not exist" containerID="ca8ffc54a67427ca6ce660375c7eb7e0a20db5e1fbff4154be815db5c27f7353" Sep 10 12:05:35.651431 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.651382 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8ffc54a67427ca6ce660375c7eb7e0a20db5e1fbff4154be815db5c27f7353"} err="failed to get container status \"ca8ffc54a67427ca6ce660375c7eb7e0a20db5e1fbff4154be815db5c27f7353\": rpc error: code = NotFound desc = could not find container \"ca8ffc54a67427ca6ce660375c7eb7e0a20db5e1fbff4154be815db5c27f7353\": container with ID starting with ca8ffc54a67427ca6ce660375c7eb7e0a20db5e1fbff4154be815db5c27f7353 not found: ID does not exist" Sep 10 12:05:35.683831 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.679200 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj"] Sep 10 12:05:35.687056 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:35.687026 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7a26e68a-openshift-default-ff7cb8bb4-57kwj"] Sep 10 12:05:36.630202 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:36.630160 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" path="/var/lib/kubelet/pods/50cc736f-12d1-4505-8611-5ecda7405a2f/volumes" Sep 10 12:05:54.739740 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:54.739701 2582 generic.go:356] "Generic (PLEG): container finished" podID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerID="1ae34138c88f378b1f25e5fe8a86eec474c37343c7b6c0ed86b15408bfaa5c07" exitCode=0 Sep 10 12:05:54.740295 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:54.739769 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" event={"ID":"6a8e1df5-54f9-4321-803a-6b40a3efda28","Type":"ContainerDied","Data":"1ae34138c88f378b1f25e5fe8a86eec474c37343c7b6c0ed86b15408bfaa5c07"} Sep 10 12:05:54.740365 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:54.740328 2582 scope.go:117] "RemoveContainer" containerID="1ae34138c88f378b1f25e5fe8a86eec474c37343c7b6c0ed86b15408bfaa5c07" Sep 10 12:05:54.741504 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:54.741474 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:54.741611 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:54.741556 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:54.741611 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:54.741601 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:05:55.366779 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:55.366741 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:05:55.746702 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:55.746670 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" event={"ID":"6a8e1df5-54f9-4321-803a-6b40a3efda28","Type":"ContainerStarted","Data":"675dab35befaf602d7a370d4fa4fac7d6d589ba0dd7499497a75b78dcf16a0dd"} Sep 10 12:05:56.277584 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:56.277550 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:05:56.277743 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:56.277724 2582 patch_prober.go:28] interesting pod/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Sep 10 12:05:56.277808 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:56.277769 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" podUID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Sep 10 12:05:57.277693 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:57.277653 2582 patch_prober.go:28] interesting pod/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Sep 10 12:05:57.278201 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:57.277728 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" podUID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Sep 10 12:05:58.276995 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:58.276962 2582 patch_prober.go:28] interesting pod/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Sep 10 12:05:58.277212 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:58.277023 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" podUID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Sep 10 12:05:59.123006 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.122968 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr"] Sep 10 12:05:59.124196 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.124173 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" Sep 10 12:05:59.124348 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.124326 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" Sep 10 12:05:59.124459 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.124368 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" Sep 10 12:05:59.124459 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.124378 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" Sep 10 12:05:59.124541 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.124489 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" Sep 10 12:05:59.124541 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.124500 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="50cc736f-12d1-4505-8611-5ecda7405a2f" containerName="istio-proxy" Sep 10 12:05:59.127781 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.127766 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr" Sep 10 12:05:59.130675 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.130657 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:05:59.130774 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.130658 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:05:59.139295 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.139274 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr"] Sep 10 12:05:59.230159 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.230109 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b93455-52c6-495c-b973-4f705b98ad87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-17efa977-khtjr\" (UID: \"21b93455-52c6-495c-b973-4f705b98ad87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr" Sep 10 12:05:59.277379 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.277351 2582 patch_prober.go:28] interesting pod/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Sep 10 12:05:59.277510 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.277401 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" podUID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Sep 10 12:05:59.331208 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.331137 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b93455-52c6-495c-b973-4f705b98ad87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-17efa977-khtjr\" (UID: \"21b93455-52c6-495c-b973-4f705b98ad87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr" Sep 10 12:05:59.331767 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.331747 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b93455-52c6-495c-b973-4f705b98ad87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-17efa977-khtjr\" (UID: \"21b93455-52c6-495c-b973-4f705b98ad87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr" Sep 10 12:05:59.439233 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.439206 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr" Sep 10 12:05:59.729799 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.729716 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr"] Sep 10 12:05:59.733503 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:05:59.733477 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b93455_52c6_495c_b973_4f705b98ad87.slice/crio-839294d91041eb763c7d0ee91de5fe3b09ffca23e7566e0227d7b174d4a65279 WatchSource:0}: Error finding container 839294d91041eb763c7d0ee91de5fe3b09ffca23e7566e0227d7b174d4a65279: Status 404 returned error can't find the container with id 839294d91041eb763c7d0ee91de5fe3b09ffca23e7566e0227d7b174d4a65279 Sep 10 12:05:59.768727 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:05:59.768693 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr" event={"ID":"21b93455-52c6-495c-b973-4f705b98ad87","Type":"ContainerStarted","Data":"839294d91041eb763c7d0ee91de5fe3b09ffca23e7566e0227d7b174d4a65279"} Sep 10 12:06:00.277094 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:00.277017 2582 patch_prober.go:28] interesting pod/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Sep 10 12:06:00.277094 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:00.277073 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" podUID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Sep 10 12:06:00.776767 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:00.776728 2582 generic.go:356] "Generic (PLEG): container finished" podID="21b93455-52c6-495c-b973-4f705b98ad87" containerID="e3a2149ad972c3a2f7b131240ae4ff81b472b16c8aa31e6122fc2dd9d37ea05b" exitCode=0 Sep 10 12:06:00.776967 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:00.776812 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr" event={"ID":"21b93455-52c6-495c-b973-4f705b98ad87","Type":"ContainerDied","Data":"e3a2149ad972c3a2f7b131240ae4ff81b472b16c8aa31e6122fc2dd9d37ea05b"} Sep 10 12:06:01.280817 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:01.280786 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:06:01.281317 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:01.281181 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:06:01.281803 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:01.281784 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:06:01.924599 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:01.924576 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr" Sep 10 12:06:01.954725 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:01.954693 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b93455-52c6-495c-b973-4f705b98ad87-maas-api-service-ca\") pod \"21b93455-52c6-495c-b973-4f705b98ad87\" (UID: \"21b93455-52c6-495c-b973-4f705b98ad87\") " Sep 10 12:06:01.955121 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:01.955093 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b93455-52c6-495c-b973-4f705b98ad87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "21b93455-52c6-495c-b973-4f705b98ad87" (UID: "21b93455-52c6-495c-b973-4f705b98ad87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:06:02.055917 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:02.055838 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b93455-52c6-495c-b973-4f705b98ad87-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:02.788604 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:02.788565 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr" event={"ID":"21b93455-52c6-495c-b973-4f705b98ad87","Type":"ContainerDied","Data":"839294d91041eb763c7d0ee91de5fe3b09ffca23e7566e0227d7b174d4a65279"} Sep 10 12:06:02.788973 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:02.788627 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="839294d91041eb763c7d0ee91de5fe3b09ffca23e7566e0227d7b174d4a65279" Sep 10 12:06:02.788973 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:02.788586 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr" Sep 10 12:06:05.949393 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:05.949354 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr"] Sep 10 12:06:05.951782 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:05.951759 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-17efa977-khtjr"] Sep 10 12:06:06.097667 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.097626 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw"] Sep 10 12:06:06.098515 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.098464 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" podUID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerName="istio-proxy" containerID="cri-o://675dab35befaf602d7a370d4fa4fac7d6d589ba0dd7499497a75b78dcf16a0dd" gracePeriod=30 Sep 10 12:06:06.630289 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.630256 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b93455-52c6-495c-b973-4f705b98ad87" path="/var/lib/kubelet/pods/21b93455-52c6-495c-b973-4f705b98ad87/volumes" Sep 10 12:06:06.852056 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.852026 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t"] Sep 10 12:06:06.852579 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.852556 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b93455-52c6-495c-b973-4f705b98ad87" containerName="revoke-keys" Sep 10 12:06:06.852579 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.852581 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b93455-52c6-495c-b973-4f705b98ad87" containerName="revoke-keys" Sep 10 12:06:06.852728 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.852662 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b93455-52c6-495c-b973-4f705b98ad87" containerName="revoke-keys" Sep 10 12:06:06.856156 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.856123 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:06.858706 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.858685 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-eb1d9122-openshift-default-dockercfg-nbq8m\"" Sep 10 12:06:06.868484 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.868462 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t"] Sep 10 12:06:06.902633 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.902550 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-data\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:06.902633 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.902585 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqfw8\" (UniqueName: \"kubernetes.io/projected/403e16f3-9bc0-4e40-8622-7c7c82409c3e-kube-api-access-bqfw8\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:06.902633 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.902607 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istiod-ca-cert\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:06.902896 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.902669 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-workload-certs\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:06.902896 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.902735 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-envoy\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:06.902896 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.902775 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-credential-socket\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:06.902896 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.902802 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-podinfo\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:06.902896 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.902858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-token\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:06.903082 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:06.902924 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-workload-socket\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.003578 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.003547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-token\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.004010 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.003585 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-workload-socket\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.004010 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.003646 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-data\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.004010 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.003665 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqfw8\" (UniqueName: \"kubernetes.io/projected/403e16f3-9bc0-4e40-8622-7c7c82409c3e-kube-api-access-bqfw8\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.004010 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.003684 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istiod-ca-cert\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.004010 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.003713 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-workload-certs\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.004010 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.003757 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-envoy\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.004010 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.003793 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-credential-socket\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.004010 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.003818 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-podinfo\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.004469 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.004115 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-data\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.004469 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.004190 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-workload-socket\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.004469 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.004317 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-workload-certs\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.004469 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.004461 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-credential-socket\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.004604 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.004567 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istiod-ca-cert\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.006380 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.006351 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-envoy\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.006534 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.006519 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-podinfo\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.012795 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.012772 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-token\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.012999 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.012979 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqfw8\" (UniqueName: \"kubernetes.io/projected/403e16f3-9bc0-4e40-8622-7c7c82409c3e-kube-api-access-bqfw8\") pod \"e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.166066 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.165983 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:07.464621 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.464595 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t"] Sep 10 12:06:07.465334 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:06:07.465309 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403e16f3_9bc0_4e40_8622_7c7c82409c3e.slice/crio-8b673b91c2a4e27b2fd40b4e2514ac72814f28196c89801a3c9ada2e7b3945e3 WatchSource:0}: Error finding container 8b673b91c2a4e27b2fd40b4e2514ac72814f28196c89801a3c9ada2e7b3945e3: Status 404 returned error can't find the container with id 8b673b91c2a4e27b2fd40b4e2514ac72814f28196c89801a3c9ada2e7b3945e3 Sep 10 12:06:07.467607 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.467568 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:06:07.467681 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.467657 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:06:07.467718 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.467705 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:06:07.820734 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.820633 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" event={"ID":"403e16f3-9bc0-4e40-8622-7c7c82409c3e","Type":"ContainerStarted","Data":"e34cabf5dd3c493047d79a071bd95fc6dc7c6ab380d37faa439c732400356d63"} Sep 10 12:06:07.820734 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.820680 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" event={"ID":"403e16f3-9bc0-4e40-8622-7c7c82409c3e","Type":"ContainerStarted","Data":"8b673b91c2a4e27b2fd40b4e2514ac72814f28196c89801a3c9ada2e7b3945e3"} Sep 10 12:06:07.845276 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:07.845222 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" podStartSLOduration=1.845206012 podStartE2EDuration="1.845206012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:06:07.843130491 +0000 UTC m=+875.722519923" watchObservedRunningTime="2026-09-10 12:06:07.845206012 +0000 UTC m=+875.724595443" Sep 10 12:06:08.166269 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:08.166164 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:08.171124 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:08.171096 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:08.827046 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:08.827013 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:08.828100 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:08.828080 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:11.367030 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.366967 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:06:11.458419 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.458387 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-envoy\") pod \"6a8e1df5-54f9-4321-803a-6b40a3efda28\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " Sep 10 12:06:11.458595 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.458435 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-workload-socket\") pod \"6a8e1df5-54f9-4321-803a-6b40a3efda28\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " Sep 10 12:06:11.458595 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.458465 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-workload-certs\") pod \"6a8e1df5-54f9-4321-803a-6b40a3efda28\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " Sep 10 12:06:11.458595 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.458534 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-token\") pod \"6a8e1df5-54f9-4321-803a-6b40a3efda28\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " Sep 10 12:06:11.458720 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.458617 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-data\") pod \"6a8e1df5-54f9-4321-803a-6b40a3efda28\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " Sep 10 12:06:11.458720 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.458644 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csbl2\" (UniqueName: \"kubernetes.io/projected/6a8e1df5-54f9-4321-803a-6b40a3efda28-kube-api-access-csbl2\") pod \"6a8e1df5-54f9-4321-803a-6b40a3efda28\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " Sep 10 12:06:11.458720 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.458687 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-credential-socket\") pod \"6a8e1df5-54f9-4321-803a-6b40a3efda28\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " Sep 10 12:06:11.458720 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.458717 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-podinfo\") pod \"6a8e1df5-54f9-4321-803a-6b40a3efda28\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " Sep 10 12:06:11.458929 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.458741 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e1df5-54f9-4321-803a-6b40a3efda28-istiod-ca-cert\") pod \"6a8e1df5-54f9-4321-803a-6b40a3efda28\" (UID: \"6a8e1df5-54f9-4321-803a-6b40a3efda28\") " Sep 10 12:06:11.458929 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.458739 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6a8e1df5-54f9-4321-803a-6b40a3efda28" (UID: "6a8e1df5-54f9-4321-803a-6b40a3efda28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:06:11.458929 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.458805 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6a8e1df5-54f9-4321-803a-6b40a3efda28" (UID: "6a8e1df5-54f9-4321-803a-6b40a3efda28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:06:11.459188 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.458958 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-data" (OuterVolumeSpecName: "istio-data") pod "6a8e1df5-54f9-4321-803a-6b40a3efda28" (UID: "6a8e1df5-54f9-4321-803a-6b40a3efda28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:06:11.459188 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.458986 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6a8e1df5-54f9-4321-803a-6b40a3efda28" (UID: "6a8e1df5-54f9-4321-803a-6b40a3efda28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:06:11.459188 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.459176 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:11.459359 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.459195 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:11.459359 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.459208 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:11.459359 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.459221 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:11.459359 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.459256 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8e1df5-54f9-4321-803a-6b40a3efda28-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6a8e1df5-54f9-4321-803a-6b40a3efda28" (UID: "6a8e1df5-54f9-4321-803a-6b40a3efda28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:06:11.460894 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.460872 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6a8e1df5-54f9-4321-803a-6b40a3efda28" (UID: "6a8e1df5-54f9-4321-803a-6b40a3efda28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:06:11.460985 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.460933 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-token" (OuterVolumeSpecName: "istio-token") pod "6a8e1df5-54f9-4321-803a-6b40a3efda28" (UID: "6a8e1df5-54f9-4321-803a-6b40a3efda28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:06:11.461033 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.460998 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6a8e1df5-54f9-4321-803a-6b40a3efda28" (UID: "6a8e1df5-54f9-4321-803a-6b40a3efda28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:06:11.461195 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.461175 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8e1df5-54f9-4321-803a-6b40a3efda28-kube-api-access-csbl2" (OuterVolumeSpecName: "kube-api-access-csbl2") pod "6a8e1df5-54f9-4321-803a-6b40a3efda28" (UID: "6a8e1df5-54f9-4321-803a-6b40a3efda28"). InnerVolumeSpecName "kube-api-access-csbl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:06:11.560079 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.560041 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csbl2\" (UniqueName: \"kubernetes.io/projected/6a8e1df5-54f9-4321-803a-6b40a3efda28-kube-api-access-csbl2\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:11.560079 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.560074 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:11.560079 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.560083 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a8e1df5-54f9-4321-803a-6b40a3efda28-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:11.560335 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.560092 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:11.560335 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.560099 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a8e1df5-54f9-4321-803a-6b40a3efda28-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:11.847244 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.847215 2582 generic.go:356] "Generic (PLEG): container finished" podID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerID="675dab35befaf602d7a370d4fa4fac7d6d589ba0dd7499497a75b78dcf16a0dd" exitCode=0 Sep 10 12:06:11.847425 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.847264 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" Sep 10 12:06:11.847425 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.847303 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" event={"ID":"6a8e1df5-54f9-4321-803a-6b40a3efda28","Type":"ContainerDied","Data":"675dab35befaf602d7a370d4fa4fac7d6d589ba0dd7499497a75b78dcf16a0dd"} Sep 10 12:06:11.847425 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.847359 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw" event={"ID":"6a8e1df5-54f9-4321-803a-6b40a3efda28","Type":"ContainerDied","Data":"ee12776e54c99c690ae635b9025a13d27c815fab3a2835d659b785071013065e"} Sep 10 12:06:11.847425 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.847388 2582 scope.go:117] "RemoveContainer" containerID="675dab35befaf602d7a370d4fa4fac7d6d589ba0dd7499497a75b78dcf16a0dd" Sep 10 12:06:11.857061 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.857044 2582 scope.go:117] "RemoveContainer" containerID="1ae34138c88f378b1f25e5fe8a86eec474c37343c7b6c0ed86b15408bfaa5c07" Sep 10 12:06:11.865504 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.865488 2582 scope.go:117] "RemoveContainer" containerID="675dab35befaf602d7a370d4fa4fac7d6d589ba0dd7499497a75b78dcf16a0dd" Sep 10 12:06:11.865751 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:06:11.865735 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675dab35befaf602d7a370d4fa4fac7d6d589ba0dd7499497a75b78dcf16a0dd\": container with ID starting with 675dab35befaf602d7a370d4fa4fac7d6d589ba0dd7499497a75b78dcf16a0dd not found: ID does not exist" containerID="675dab35befaf602d7a370d4fa4fac7d6d589ba0dd7499497a75b78dcf16a0dd" Sep 10 12:06:11.865802 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.865757 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675dab35befaf602d7a370d4fa4fac7d6d589ba0dd7499497a75b78dcf16a0dd"} err="failed to get container status \"675dab35befaf602d7a370d4fa4fac7d6d589ba0dd7499497a75b78dcf16a0dd\": rpc error: code = NotFound desc = could not find container \"675dab35befaf602d7a370d4fa4fac7d6d589ba0dd7499497a75b78dcf16a0dd\": container with ID starting with 675dab35befaf602d7a370d4fa4fac7d6d589ba0dd7499497a75b78dcf16a0dd not found: ID does not exist" Sep 10 12:06:11.865802 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.865772 2582 scope.go:117] "RemoveContainer" containerID="1ae34138c88f378b1f25e5fe8a86eec474c37343c7b6c0ed86b15408bfaa5c07" Sep 10 12:06:11.866002 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:06:11.865988 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae34138c88f378b1f25e5fe8a86eec474c37343c7b6c0ed86b15408bfaa5c07\": container with ID starting with 1ae34138c88f378b1f25e5fe8a86eec474c37343c7b6c0ed86b15408bfaa5c07 not found: ID does not exist" containerID="1ae34138c88f378b1f25e5fe8a86eec474c37343c7b6c0ed86b15408bfaa5c07" Sep 10 12:06:11.866039 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.866006 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae34138c88f378b1f25e5fe8a86eec474c37343c7b6c0ed86b15408bfaa5c07"} err="failed to get container status \"1ae34138c88f378b1f25e5fe8a86eec474c37343c7b6c0ed86b15408bfaa5c07\": rpc error: code = NotFound desc = could not find container \"1ae34138c88f378b1f25e5fe8a86eec474c37343c7b6c0ed86b15408bfaa5c07\": container with ID starting with 1ae34138c88f378b1f25e5fe8a86eec474c37343c7b6c0ed86b15408bfaa5c07 not found: ID does not exist" Sep 10 12:06:11.870954 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.870932 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw"] Sep 10 12:06:11.876632 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:11.876611 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-17efa977-gw-openshift-default-6b787c788-945xw"] Sep 10 12:06:12.630670 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:12.630628 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a8e1df5-54f9-4321-803a-6b40a3efda28" path="/var/lib/kubelet/pods/6a8e1df5-54f9-4321-803a-6b40a3efda28/volumes" Sep 10 12:06:14.667593 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:14.667548 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:06:21.977988 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:06:21.977957 2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/systemd-tmpfiles-clean.service\": RecentStats: unable to find data in memory cache]" Sep 10 12:06:24.869409 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:24.869319 2582 patch_prober.go:28] interesting pod/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 12:06:24.869789 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:24.869425 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 12:06:25.926358 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:25.926323 2582 generic.go:356] "Generic (PLEG): container finished" podID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerID="e34cabf5dd3c493047d79a071bd95fc6dc7c6ab380d37faa439c732400356d63" exitCode=0 Sep 10 12:06:25.926782 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:25.926396 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" event={"ID":"403e16f3-9bc0-4e40-8622-7c7c82409c3e","Type":"ContainerDied","Data":"e34cabf5dd3c493047d79a071bd95fc6dc7c6ab380d37faa439c732400356d63"} Sep 10 12:06:25.926836 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:25.926826 2582 scope.go:117] "RemoveContainer" containerID="e34cabf5dd3c493047d79a071bd95fc6dc7c6ab380d37faa439c732400356d63" Sep 10 12:06:25.927882 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:25.927850 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:06:25.927985 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:25.927938 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:06:25.928048 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:25.927988 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:06:26.948930 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:26.948884 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" event={"ID":"403e16f3-9bc0-4e40-8622-7c7c82409c3e","Type":"ContainerStarted","Data":"ac5793d9a136c7cfa780af3c1649b760b762f69c46e4d110282b126cbb1d5826"} Sep 10 12:06:27.167249 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:27.167205 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:27.167249 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:27.167246 2582 patch_prober.go:28] interesting pod/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 12:06:27.167507 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:27.167299 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 12:06:28.167077 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:28.167035 2582 patch_prober.go:28] interesting pod/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 12:06:28.167506 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:28.167096 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 12:06:29.166791 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:29.166758 2582 patch_prober.go:28] interesting pod/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 12:06:29.166967 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:29.166827 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 12:06:30.166576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:30.166539 2582 patch_prober.go:28] interesting pod/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 12:06:30.166970 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:30.166598 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 12:06:31.167217 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:31.167179 2582 patch_prober.go:28] interesting pod/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 12:06:31.167703 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:31.167244 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 12:06:32.167218 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:32.167180 2582 patch_prober.go:28] interesting pod/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 12:06:32.167608 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:32.167238 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 12:06:32.614287 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:32.614261 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 12:06:32.634840 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:32.634818 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 12:06:33.170201 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:33.170174 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:33.170591 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:33.170469 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:33.171249 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:33.171233 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:34.658958 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:34.658925 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:06:35.044008 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.043972 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h"] Sep 10 12:06:35.044438 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.044421 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerName="istio-proxy" Sep 10 12:06:35.044438 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.044438 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerName="istio-proxy" Sep 10 12:06:35.044588 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.044453 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerName="istio-proxy" Sep 10 12:06:35.044588 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.044459 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerName="istio-proxy" Sep 10 12:06:35.044588 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.044529 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerName="istio-proxy" Sep 10 12:06:35.044588 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.044540 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8e1df5-54f9-4321-803a-6b40a3efda28" containerName="istio-proxy" Sep 10 12:06:35.049076 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.049058 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h" Sep 10 12:06:35.052067 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.052045 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:06:35.052199 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.052045 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:06:35.056232 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.056202 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h"] Sep 10 12:06:35.103400 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.103364 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07a9edb-2ee0-4c15-9657-42c69a59b3d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h\" (UID: \"f07a9edb-2ee0-4c15-9657-42c69a59b3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h" Sep 10 12:06:35.204667 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.204630 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07a9edb-2ee0-4c15-9657-42c69a59b3d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h\" (UID: \"f07a9edb-2ee0-4c15-9657-42c69a59b3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h" Sep 10 12:06:35.205269 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.205250 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07a9edb-2ee0-4c15-9657-42c69a59b3d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h\" (UID: \"f07a9edb-2ee0-4c15-9657-42c69a59b3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h" Sep 10 12:06:35.361225 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.361111 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h" Sep 10 12:06:35.638563 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.638537 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h"] Sep 10 12:06:35.642113 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:06:35.642087 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07a9edb_2ee0_4c15_9657_42c69a59b3d6.slice/crio-4e7f32491f00dc3dd34eae8e919c58dd40081634a711466a671ef58d0b44ad83 WatchSource:0}: Error finding container 4e7f32491f00dc3dd34eae8e919c58dd40081634a711466a671ef58d0b44ad83: Status 404 returned error can't find the container with id 4e7f32491f00dc3dd34eae8e919c58dd40081634a711466a671ef58d0b44ad83 Sep 10 12:06:35.999352 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:35.999273 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h" event={"ID":"f07a9edb-2ee0-4c15-9657-42c69a59b3d6","Type":"ContainerStarted","Data":"4e7f32491f00dc3dd34eae8e919c58dd40081634a711466a671ef58d0b44ad83"} Sep 10 12:06:37.006833 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:37.006316 2582 generic.go:356] "Generic (PLEG): container finished" podID="f07a9edb-2ee0-4c15-9657-42c69a59b3d6" containerID="44eadfa1d342466102aeff9dbb5b7121490db9335157e1d78e132e55f0032ebe" exitCode=0 Sep 10 12:06:37.006833 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:37.006395 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h" event={"ID":"f07a9edb-2ee0-4c15-9657-42c69a59b3d6","Type":"ContainerDied","Data":"44eadfa1d342466102aeff9dbb5b7121490db9335157e1d78e132e55f0032ebe"} Sep 10 12:06:38.151032 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:38.150999 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h" Sep 10 12:06:38.236126 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:38.236085 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07a9edb-2ee0-4c15-9657-42c69a59b3d6-maas-api-service-ca\") pod \"f07a9edb-2ee0-4c15-9657-42c69a59b3d6\" (UID: \"f07a9edb-2ee0-4c15-9657-42c69a59b3d6\") " Sep 10 12:06:38.236528 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:38.236501 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07a9edb-2ee0-4c15-9657-42c69a59b3d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f07a9edb-2ee0-4c15-9657-42c69a59b3d6" (UID: "f07a9edb-2ee0-4c15-9657-42c69a59b3d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:06:38.337630 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:38.337545 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07a9edb-2ee0-4c15-9657-42c69a59b3d6-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:39.019060 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:39.019027 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h" Sep 10 12:06:39.019285 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:39.019063 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h" event={"ID":"f07a9edb-2ee0-4c15-9657-42c69a59b3d6","Type":"ContainerDied","Data":"4e7f32491f00dc3dd34eae8e919c58dd40081634a711466a671ef58d0b44ad83"} Sep 10 12:06:39.019285 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:39.019117 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e7f32491f00dc3dd34eae8e919c58dd40081634a711466a671ef58d0b44ad83" Sep 10 12:06:41.582419 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:41.582385 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h"] Sep 10 12:06:41.585666 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:41.585645 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb1d9122-jsn9h"] Sep 10 12:06:42.636063 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:42.636028 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f07a9edb-2ee0-4c15-9657-42c69a59b3d6" path="/var/lib/kubelet/pods/f07a9edb-2ee0-4c15-9657-42c69a59b3d6/volumes" Sep 10 12:06:42.833938 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:42.833909 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t"] Sep 10 12:06:42.834180 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:42.834125 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" containerID="cri-o://ac5793d9a136c7cfa780af3c1649b760b762f69c46e4d110282b126cbb1d5826" gracePeriod=30 Sep 10 12:06:43.649521 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.649492 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp"] Sep 10 12:06:43.650072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.650055 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f07a9edb-2ee0-4c15-9657-42c69a59b3d6" containerName="revoke-keys" Sep 10 12:06:43.650181 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.650074 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f07a9edb-2ee0-4c15-9657-42c69a59b3d6" containerName="revoke-keys" Sep 10 12:06:43.650181 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.650176 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f07a9edb-2ee0-4c15-9657-42c69a59b3d6" containerName="revoke-keys" Sep 10 12:06:43.653794 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.653773 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.656504 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.656483 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4d5fbdf0-openshift-default-dockercfg-hfc6r\"" Sep 10 12:06:43.665455 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.665437 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp"] Sep 10 12:06:43.792261 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.792228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-workload-certs\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.792261 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.792270 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-token\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.792543 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.792301 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-podinfo\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.792543 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.792393 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwz7t\" (UniqueName: \"kubernetes.io/projected/fcba9edd-88c8-4552-b985-c9c7bdb657ed-kube-api-access-kwz7t\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.792543 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.792452 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istiod-ca-cert\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.792543 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.792481 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-workload-socket\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.792543 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.792506 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-envoy\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.792760 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.792583 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-data\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.792760 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.792642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-credential-socket\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.893543 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.893511 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-credential-socket\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.893748 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.893552 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-workload-certs\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.893748 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.893575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-token\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.893748 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.893616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-podinfo\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.893748 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.893665 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwz7t\" (UniqueName: \"kubernetes.io/projected/fcba9edd-88c8-4552-b985-c9c7bdb657ed-kube-api-access-kwz7t\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.893748 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.893707 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istiod-ca-cert\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.893748 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.893733 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-workload-socket\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.894077 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.893753 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-envoy\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.894077 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.893804 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-data\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.894077 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.894026 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-credential-socket\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.894077 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.894048 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-workload-certs\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.894368 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.894113 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-workload-socket\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.894368 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.894250 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-data\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.894768 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.894738 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istiod-ca-cert\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.896806 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.896783 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-envoy\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.899677 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.899617 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-podinfo\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.902542 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.902513 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-token\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.902656 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.902626 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwz7t\" (UniqueName: \"kubernetes.io/projected/fcba9edd-88c8-4552-b985-c9c7bdb657ed-kube-api-access-kwz7t\") pod \"e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:43.965508 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:43.965469 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:44.248018 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:44.247988 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp"] Sep 10 12:06:44.250683 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:06:44.250657 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcba9edd_88c8_4552_b985_c9c7bdb657ed.slice/crio-641946b7a3ca065d68dc52ee1fb610705a318ddea4470578bd62f117723bdc1e WatchSource:0}: Error finding container 641946b7a3ca065d68dc52ee1fb610705a318ddea4470578bd62f117723bdc1e: Status 404 returned error can't find the container with id 641946b7a3ca065d68dc52ee1fb610705a318ddea4470578bd62f117723bdc1e Sep 10 12:06:44.253280 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:44.253245 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:06:44.253385 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:44.253312 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:06:44.253385 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:44.253342 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:06:45.052556 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:45.052519 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" event={"ID":"fcba9edd-88c8-4552-b985-c9c7bdb657ed","Type":"ContainerStarted","Data":"0a2fb45fcd7361f66cff31a406c7417498d0d027cb43dc3f02533ddf4b113da0"} Sep 10 12:06:45.053041 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:45.052585 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" event={"ID":"fcba9edd-88c8-4552-b985-c9c7bdb657ed","Type":"ContainerStarted","Data":"641946b7a3ca065d68dc52ee1fb610705a318ddea4470578bd62f117723bdc1e"} Sep 10 12:06:45.077685 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:45.077638 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" podStartSLOduration=2.077622092 podStartE2EDuration="2.077622092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:06:45.075001389 +0000 UTC m=+912.954390826" watchObservedRunningTime="2026-09-10 12:06:45.077622092 +0000 UTC m=+912.957011524" Sep 10 12:06:45.966642 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:45.966608 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:45.971460 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:45.971434 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:46.063894 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:46.063857 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:46.064798 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:46.064776 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:06:48.078705 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.078679 2582 generic.go:356] "Generic (PLEG): container finished" podID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerID="ac5793d9a136c7cfa780af3c1649b760b762f69c46e4d110282b126cbb1d5826" exitCode=0 Sep 10 12:06:48.079056 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.078746 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" event={"ID":"403e16f3-9bc0-4e40-8622-7c7c82409c3e","Type":"ContainerDied","Data":"ac5793d9a136c7cfa780af3c1649b760b762f69c46e4d110282b126cbb1d5826"} Sep 10 12:06:48.079056 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.078804 2582 scope.go:117] "RemoveContainer" containerID="e34cabf5dd3c493047d79a071bd95fc6dc7c6ab380d37faa439c732400356d63" Sep 10 12:06:48.100574 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:06:48.100544 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34cabf5dd3c493047d79a071bd95fc6dc7c6ab380d37faa439c732400356d63\": container with ID starting with e34cabf5dd3c493047d79a071bd95fc6dc7c6ab380d37faa439c732400356d63 not found: ID does not exist" containerID="e34cabf5dd3c493047d79a071bd95fc6dc7c6ab380d37faa439c732400356d63" Sep 10 12:06:48.104053 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.104036 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:48.235792 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.235754 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqfw8\" (UniqueName: \"kubernetes.io/projected/403e16f3-9bc0-4e40-8622-7c7c82409c3e-kube-api-access-bqfw8\") pod \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " Sep 10 12:06:48.235987 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.235853 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-workload-certs\") pod \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " Sep 10 12:06:48.235987 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.235896 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-podinfo\") pod \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " Sep 10 12:06:48.235987 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.235928 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-credential-socket\") pod \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " Sep 10 12:06:48.235987 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.235953 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-envoy\") pod \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " Sep 10 12:06:48.235987 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.235978 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-workload-socket\") pod \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " Sep 10 12:06:48.236293 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.236012 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-data\") pod \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " Sep 10 12:06:48.236293 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.236045 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istiod-ca-cert\") pod \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " Sep 10 12:06:48.236293 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.236112 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-token\") pod \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\" (UID: \"403e16f3-9bc0-4e40-8622-7c7c82409c3e\") " Sep 10 12:06:48.236293 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.236184 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "403e16f3-9bc0-4e40-8622-7c7c82409c3e" (UID: "403e16f3-9bc0-4e40-8622-7c7c82409c3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:06:48.236293 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.236235 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "403e16f3-9bc0-4e40-8622-7c7c82409c3e" (UID: "403e16f3-9bc0-4e40-8622-7c7c82409c3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:06:48.236293 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.236268 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "403e16f3-9bc0-4e40-8622-7c7c82409c3e" (UID: "403e16f3-9bc0-4e40-8622-7c7c82409c3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:06:48.236631 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.236404 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-data" (OuterVolumeSpecName: "istio-data") pod "403e16f3-9bc0-4e40-8622-7c7c82409c3e" (UID: "403e16f3-9bc0-4e40-8622-7c7c82409c3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:06:48.236631 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.236585 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:48.236631 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.236607 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:48.236631 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.236613 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "403e16f3-9bc0-4e40-8622-7c7c82409c3e" (UID: "403e16f3-9bc0-4e40-8622-7c7c82409c3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:06:48.236631 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.236622 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:48.237073 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.236656 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:48.238483 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.238454 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "403e16f3-9bc0-4e40-8622-7c7c82409c3e" (UID: "403e16f3-9bc0-4e40-8622-7c7c82409c3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:06:48.238597 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.238479 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-token" (OuterVolumeSpecName: "istio-token") pod "403e16f3-9bc0-4e40-8622-7c7c82409c3e" (UID: "403e16f3-9bc0-4e40-8622-7c7c82409c3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:06:48.238597 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.238500 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "403e16f3-9bc0-4e40-8622-7c7c82409c3e" (UID: "403e16f3-9bc0-4e40-8622-7c7c82409c3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:06:48.238597 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.238514 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403e16f3-9bc0-4e40-8622-7c7c82409c3e-kube-api-access-bqfw8" (OuterVolumeSpecName: "kube-api-access-bqfw8") pod "403e16f3-9bc0-4e40-8622-7c7c82409c3e" (UID: "403e16f3-9bc0-4e40-8622-7c7c82409c3e"). InnerVolumeSpecName "kube-api-access-bqfw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:06:48.337449 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.337415 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:48.337449 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.337443 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:48.337449 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.337452 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqfw8\" (UniqueName: \"kubernetes.io/projected/403e16f3-9bc0-4e40-8622-7c7c82409c3e-kube-api-access-bqfw8\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:48.338138 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.337461 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:48.338138 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:48.337470 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403e16f3-9bc0-4e40-8622-7c7c82409c3e-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:06:49.085495 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:49.085463 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" event={"ID":"403e16f3-9bc0-4e40-8622-7c7c82409c3e","Type":"ContainerDied","Data":"8b673b91c2a4e27b2fd40b4e2514ac72814f28196c89801a3c9ada2e7b3945e3"} Sep 10 12:06:49.085981 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:49.085515 2582 scope.go:117] "RemoveContainer" containerID="ac5793d9a136c7cfa780af3c1649b760b762f69c46e4d110282b126cbb1d5826" Sep 10 12:06:49.085981 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:49.085477 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t" Sep 10 12:06:49.105391 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:49.105367 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t"] Sep 10 12:06:49.110510 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:49.110486 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-eb1d9122-openshift-default-d6867d7b4-hn97t"] Sep 10 12:06:50.629896 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:50.629857 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" path="/var/lib/kubelet/pods/403e16f3-9bc0-4e40-8622-7c7c82409c3e/volumes" Sep 10 12:06:52.859922 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:06:52.859890 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:07:02.105303 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:02.105265 2582 patch_prober.go:28] interesting pod/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 12:07:02.105772 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:02.105351 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 12:07:04.175646 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:04.175529 2582 generic.go:356] "Generic (PLEG): container finished" podID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerID="0a2fb45fcd7361f66cff31a406c7417498d0d027cb43dc3f02533ddf4b113da0" exitCode=0 Sep 10 12:07:04.175646 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:04.175611 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" event={"ID":"fcba9edd-88c8-4552-b985-c9c7bdb657ed","Type":"ContainerDied","Data":"0a2fb45fcd7361f66cff31a406c7417498d0d027cb43dc3f02533ddf4b113da0"} Sep 10 12:07:04.176196 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:04.176020 2582 scope.go:117] "RemoveContainer" containerID="0a2fb45fcd7361f66cff31a406c7417498d0d027cb43dc3f02533ddf4b113da0" Sep 10 12:07:04.177117 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:04.177089 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:07:04.177223 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:04.177171 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:07:04.177223 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:04.177204 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:07:05.183321 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:05.183285 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" event={"ID":"fcba9edd-88c8-4552-b985-c9c7bdb657ed","Type":"ContainerStarted","Data":"f335e9aa8ee6908719ef4a8b70f0985feacf458eea732cdbc3edc1f2bf4a3270"} Sep 10 12:07:05.966648 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:05.966612 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:07:05.966843 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:05.966823 2582 patch_prober.go:28] interesting pod/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 12:07:05.966890 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:05.966869 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 12:07:06.965985 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:06.965951 2582 patch_prober.go:28] interesting pod/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 12:07:06.966400 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:06.966008 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 12:07:07.966587 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:07.966551 2582 patch_prober.go:28] interesting pod/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 12:07:07.966985 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:07.966614 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 12:07:08.966793 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:08.966754 2582 patch_prober.go:28] interesting pod/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 12:07:08.967197 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:08.966813 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 12:07:09.966201 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:09.966135 2582 patch_prober.go:28] interesting pod/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 10 12:07:09.966379 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:09.966226 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 10 12:07:10.969723 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:10.969694 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:07:10.970118 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:10.969960 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:07:10.970621 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:10.970607 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:07:11.387555 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.387472 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769"] Sep 10 12:07:11.387893 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.387881 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" Sep 10 12:07:11.387975 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.387895 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" Sep 10 12:07:11.387975 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.387910 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" Sep 10 12:07:11.387975 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.387915 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" Sep 10 12:07:11.387975 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.387974 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" Sep 10 12:07:11.389940 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.389924 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769" Sep 10 12:07:11.393137 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.393112 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:07:11.393274 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.393116 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:07:11.400346 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.400327 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769"] Sep 10 12:07:11.478288 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.478260 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b832978e-d48d-4020-b236-f3c513ed4519-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769\" (UID: \"b832978e-d48d-4020-b236-f3c513ed4519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769" Sep 10 12:07:11.579084 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.579052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b832978e-d48d-4020-b236-f3c513ed4519-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769\" (UID: \"b832978e-d48d-4020-b236-f3c513ed4519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769" Sep 10 12:07:11.579704 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.579683 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b832978e-d48d-4020-b236-f3c513ed4519-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769\" (UID: \"b832978e-d48d-4020-b236-f3c513ed4519\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769" Sep 10 12:07:11.702158 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.702121 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769" Sep 10 12:07:11.971701 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:11.971675 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769"] Sep 10 12:07:11.974033 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:07:11.974007 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb832978e_d48d_4020_b236_f3c513ed4519.slice/crio-87960cc7d0aeb1a75cc9a2fe38d9bd072071fd097ad4920d27e4be3f7e287882 WatchSource:0}: Error finding container 87960cc7d0aeb1a75cc9a2fe38d9bd072071fd097ad4920d27e4be3f7e287882: Status 404 returned error can't find the container with id 87960cc7d0aeb1a75cc9a2fe38d9bd072071fd097ad4920d27e4be3f7e287882 Sep 10 12:07:12.232979 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:12.232909 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769" event={"ID":"b832978e-d48d-4020-b236-f3c513ed4519","Type":"ContainerStarted","Data":"87960cc7d0aeb1a75cc9a2fe38d9bd072071fd097ad4920d27e4be3f7e287882"} Sep 10 12:07:13.153830 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:13.153731 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:07:13.240111 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:13.240077 2582 generic.go:356] "Generic (PLEG): container finished" podID="b832978e-d48d-4020-b236-f3c513ed4519" containerID="de61d48129bae198ef503d498207d832cb97322daa6e4c8374249968c6c113fb" exitCode=0 Sep 10 12:07:13.240304 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:13.240174 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769" event={"ID":"b832978e-d48d-4020-b236-f3c513ed4519","Type":"ContainerDied","Data":"de61d48129bae198ef503d498207d832cb97322daa6e4c8374249968c6c113fb"} Sep 10 12:07:14.382405 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:14.382384 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769" Sep 10 12:07:14.511019 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:14.510929 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b832978e-d48d-4020-b236-f3c513ed4519-maas-api-service-ca\") pod \"b832978e-d48d-4020-b236-f3c513ed4519\" (UID: \"b832978e-d48d-4020-b236-f3c513ed4519\") " Sep 10 12:07:14.511398 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:14.511370 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b832978e-d48d-4020-b236-f3c513ed4519-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b832978e-d48d-4020-b236-f3c513ed4519" (UID: "b832978e-d48d-4020-b236-f3c513ed4519"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:07:14.612639 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:14.612607 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b832978e-d48d-4020-b236-f3c513ed4519-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:07:15.252550 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.252524 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769" Sep 10 12:07:15.252734 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.252563 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769" event={"ID":"b832978e-d48d-4020-b236-f3c513ed4519","Type":"ContainerDied","Data":"87960cc7d0aeb1a75cc9a2fe38d9bd072071fd097ad4920d27e4be3f7e287882"} Sep 10 12:07:15.252734 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.252608 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87960cc7d0aeb1a75cc9a2fe38d9bd072071fd097ad4920d27e4be3f7e287882" Sep 10 12:07:15.286476 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.286448 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48"] Sep 10 12:07:15.286836 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.286824 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b832978e-d48d-4020-b236-f3c513ed4519" containerName="revoke-keys" Sep 10 12:07:15.286876 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.286838 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b832978e-d48d-4020-b236-f3c513ed4519" containerName="revoke-keys" Sep 10 12:07:15.286925 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.286916 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="403e16f3-9bc0-4e40-8622-7c7c82409c3e" containerName="istio-proxy" Sep 10 12:07:15.286968 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.286929 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b832978e-d48d-4020-b236-f3c513ed4519" containerName="revoke-keys" Sep 10 12:07:15.289089 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.289060 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48" Sep 10 12:07:15.291934 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.291916 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:07:15.291934 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.291921 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:07:15.300021 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.299766 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48"] Sep 10 12:07:15.420257 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.420223 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a3d73c-5524-4fb9-9e14-d51d6974db04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7608bc-zhl48\" (UID: \"c2a3d73c-5524-4fb9-9e14-d51d6974db04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48" Sep 10 12:07:15.521476 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.521385 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a3d73c-5524-4fb9-9e14-d51d6974db04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7608bc-zhl48\" (UID: \"c2a3d73c-5524-4fb9-9e14-d51d6974db04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48" Sep 10 12:07:15.522190 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.522167 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a3d73c-5524-4fb9-9e14-d51d6974db04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7608bc-zhl48\" (UID: \"c2a3d73c-5524-4fb9-9e14-d51d6974db04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48" Sep 10 12:07:15.601184 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.601133 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48" Sep 10 12:07:15.871883 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:15.871856 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48"] Sep 10 12:07:15.873568 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:07:15.873537 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a3d73c_5524_4fb9_9e14_d51d6974db04.slice/crio-1616437fc75883d278e9023c433a9d7adec907a4dee6ed7445a43ffa86a3e03b WatchSource:0}: Error finding container 1616437fc75883d278e9023c433a9d7adec907a4dee6ed7445a43ffa86a3e03b: Status 404 returned error can't find the container with id 1616437fc75883d278e9023c433a9d7adec907a4dee6ed7445a43ffa86a3e03b Sep 10 12:07:16.259536 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:16.259501 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48" event={"ID":"c2a3d73c-5524-4fb9-9e14-d51d6974db04","Type":"ContainerStarted","Data":"c59d14f79838564931bdf4b4863b2868e92d6caf24c13092618dbf3114b211ca"} Sep 10 12:07:16.259694 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:16.259552 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48" event={"ID":"c2a3d73c-5524-4fb9-9e14-d51d6974db04","Type":"ContainerStarted","Data":"1616437fc75883d278e9023c433a9d7adec907a4dee6ed7445a43ffa86a3e03b"} Sep 10 12:07:16.275615 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:16.275572 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48" podStartSLOduration=1.275556952 podStartE2EDuration="1.275556952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:07:16.274681405 +0000 UTC m=+944.154070837" watchObservedRunningTime="2026-09-10 12:07:16.275556952 +0000 UTC m=+944.154946383" Sep 10 12:07:17.266256 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:17.266222 2582 generic.go:356] "Generic (PLEG): container finished" podID="c2a3d73c-5524-4fb9-9e14-d51d6974db04" containerID="c59d14f79838564931bdf4b4863b2868e92d6caf24c13092618dbf3114b211ca" exitCode=0 Sep 10 12:07:17.266632 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:17.266308 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48" event={"ID":"c2a3d73c-5524-4fb9-9e14-d51d6974db04","Type":"ContainerDied","Data":"c59d14f79838564931bdf4b4863b2868e92d6caf24c13092618dbf3114b211ca"} Sep 10 12:07:18.413706 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:18.413682 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48" Sep 10 12:07:18.549316 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:18.549230 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a3d73c-5524-4fb9-9e14-d51d6974db04-maas-api-service-ca\") pod \"c2a3d73c-5524-4fb9-9e14-d51d6974db04\" (UID: \"c2a3d73c-5524-4fb9-9e14-d51d6974db04\") " Sep 10 12:07:18.549539 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:18.549515 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a3d73c-5524-4fb9-9e14-d51d6974db04-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2a3d73c-5524-4fb9-9e14-d51d6974db04" (UID: "c2a3d73c-5524-4fb9-9e14-d51d6974db04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:07:18.549632 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:18.549613 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2a3d73c-5524-4fb9-9e14-d51d6974db04-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:07:18.895369 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:18.895280 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769"] Sep 10 12:07:18.898712 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:18.898682 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5fbdf0-zb769"] Sep 10 12:07:19.066277 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:19.066245 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp"] Sep 10 12:07:19.066575 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:19.066497 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerName="istio-proxy" containerID="cri-o://f335e9aa8ee6908719ef4a8b70f0985feacf458eea732cdbc3edc1f2bf4a3270" gracePeriod=30 Sep 10 12:07:19.278206 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:19.278179 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48" Sep 10 12:07:19.278371 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:19.278181 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48" event={"ID":"c2a3d73c-5524-4fb9-9e14-d51d6974db04","Type":"ContainerDied","Data":"1616437fc75883d278e9023c433a9d7adec907a4dee6ed7445a43ffa86a3e03b"} Sep 10 12:07:19.278371 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:19.278298 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1616437fc75883d278e9023c433a9d7adec907a4dee6ed7445a43ffa86a3e03b" Sep 10 12:07:20.630026 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:20.629992 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b832978e-d48d-4020-b236-f3c513ed4519" path="/var/lib/kubelet/pods/b832978e-d48d-4020-b236-f3c513ed4519/volumes" Sep 10 12:07:24.311211 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.311184 2582 generic.go:356] "Generic (PLEG): container finished" podID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerID="f335e9aa8ee6908719ef4a8b70f0985feacf458eea732cdbc3edc1f2bf4a3270" exitCode=0 Sep 10 12:07:24.311539 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.311246 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" event={"ID":"fcba9edd-88c8-4552-b985-c9c7bdb657ed","Type":"ContainerDied","Data":"f335e9aa8ee6908719ef4a8b70f0985feacf458eea732cdbc3edc1f2bf4a3270"} Sep 10 12:07:24.311539 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.311317 2582 scope.go:117] "RemoveContainer" containerID="0a2fb45fcd7361f66cff31a406c7417498d0d027cb43dc3f02533ddf4b113da0" Sep 10 12:07:24.323963 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:07:24.323940 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2fb45fcd7361f66cff31a406c7417498d0d027cb43dc3f02533ddf4b113da0\": container with ID starting with 0a2fb45fcd7361f66cff31a406c7417498d0d027cb43dc3f02533ddf4b113da0 not found: ID does not exist" containerID="0a2fb45fcd7361f66cff31a406c7417498d0d027cb43dc3f02533ddf4b113da0" Sep 10 12:07:24.325481 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.325464 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:07:24.408847 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.408775 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwz7t\" (UniqueName: \"kubernetes.io/projected/fcba9edd-88c8-4552-b985-c9c7bdb657ed-kube-api-access-kwz7t\") pod \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " Sep 10 12:07:24.408847 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.408813 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-podinfo\") pod \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " Sep 10 12:07:24.409028 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.408851 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-workload-certs\") pod \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " Sep 10 12:07:24.409028 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.408876 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-envoy\") pod \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " Sep 10 12:07:24.409028 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.408916 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-workload-socket\") pod \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " Sep 10 12:07:24.409028 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.408954 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-token\") pod \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " Sep 10 12:07:24.409028 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.408991 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istiod-ca-cert\") pod \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " Sep 10 12:07:24.409028 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.409027 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-data\") pod \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " Sep 10 12:07:24.409373 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.409087 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-credential-socket\") pod \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\" (UID: \"fcba9edd-88c8-4552-b985-c9c7bdb657ed\") " Sep 10 12:07:24.409373 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.409230 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fcba9edd-88c8-4552-b985-c9c7bdb657ed" (UID: "fcba9edd-88c8-4552-b985-c9c7bdb657ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:07:24.409373 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.409241 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fcba9edd-88c8-4552-b985-c9c7bdb657ed" (UID: "fcba9edd-88c8-4552-b985-c9c7bdb657ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:07:24.409530 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.409433 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fcba9edd-88c8-4552-b985-c9c7bdb657ed" (UID: "fcba9edd-88c8-4552-b985-c9c7bdb657ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:07:24.409761 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.409536 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fcba9edd-88c8-4552-b985-c9c7bdb657ed" (UID: "fcba9edd-88c8-4552-b985-c9c7bdb657ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:07:24.409761 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.409603 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-data" (OuterVolumeSpecName: "istio-data") pod "fcba9edd-88c8-4552-b985-c9c7bdb657ed" (UID: "fcba9edd-88c8-4552-b985-c9c7bdb657ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:07:24.409871 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.409764 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:07:24.409871 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.409787 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:07:24.409871 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.409803 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:07:24.409871 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.409817 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:07:24.409871 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.409831 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:07:24.411178 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.411128 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcba9edd-88c8-4552-b985-c9c7bdb657ed-kube-api-access-kwz7t" (OuterVolumeSpecName: "kube-api-access-kwz7t") pod "fcba9edd-88c8-4552-b985-c9c7bdb657ed" (UID: "fcba9edd-88c8-4552-b985-c9c7bdb657ed"). InnerVolumeSpecName "kube-api-access-kwz7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:07:24.411287 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.411249 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fcba9edd-88c8-4552-b985-c9c7bdb657ed" (UID: "fcba9edd-88c8-4552-b985-c9c7bdb657ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:07:24.411681 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.411659 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fcba9edd-88c8-4552-b985-c9c7bdb657ed" (UID: "fcba9edd-88c8-4552-b985-c9c7bdb657ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:07:24.411681 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.411672 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "fcba9edd-88c8-4552-b985-c9c7bdb657ed" (UID: "fcba9edd-88c8-4552-b985-c9c7bdb657ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:07:24.510593 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.510563 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:07:24.510593 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.510588 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwz7t\" (UniqueName: \"kubernetes.io/projected/fcba9edd-88c8-4552-b985-c9c7bdb657ed-kube-api-access-kwz7t\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:07:24.510593 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.510598 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:07:24.510795 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:24.510607 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcba9edd-88c8-4552-b985-c9c7bdb657ed-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:07:25.318489 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:25.318464 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" Sep 10 12:07:25.318894 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:25.318500 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp" event={"ID":"fcba9edd-88c8-4552-b985-c9c7bdb657ed","Type":"ContainerDied","Data":"641946b7a3ca065d68dc52ee1fb610705a318ddea4470578bd62f117723bdc1e"} Sep 10 12:07:25.318894 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:25.318553 2582 scope.go:117] "RemoveContainer" containerID="f335e9aa8ee6908719ef4a8b70f0985feacf458eea732cdbc3edc1f2bf4a3270" Sep 10 12:07:25.339071 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:25.339040 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp"] Sep 10 12:07:25.345751 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:25.345721 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4d5fbdf0-openshift-default-756f97ccf6-6nwpp"] Sep 10 12:07:25.668214 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:25.668113 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:07:26.574833 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:26.574796 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48"] Sep 10 12:07:26.579420 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:26.579395 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7608bc-zhl48"] Sep 10 12:07:26.631624 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:26.631595 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2a3d73c-5524-4fb9-9e14-d51d6974db04" path="/var/lib/kubelet/pods/c2a3d73c-5524-4fb9-9e14-d51d6974db04/volumes" Sep 10 12:07:26.632313 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:26.632293 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" path="/var/lib/kubelet/pods/fcba9edd-88c8-4552-b985-c9c7bdb657ed/volumes" Sep 10 12:07:46.152320 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:46.152278 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:07:58.459978 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:07:58.459897 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:08:10.477827 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.477787 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v"] Sep 10 12:08:10.478433 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.478403 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a3d73c-5524-4fb9-9e14-d51d6974db04" containerName="revoke-keys" Sep 10 12:08:10.478514 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.478436 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a3d73c-5524-4fb9-9e14-d51d6974db04" containerName="revoke-keys" Sep 10 12:08:10.478514 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.478447 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerName="istio-proxy" Sep 10 12:08:10.478514 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.478456 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerName="istio-proxy" Sep 10 12:08:10.478514 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.478481 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerName="istio-proxy" Sep 10 12:08:10.478514 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.478486 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerName="istio-proxy" Sep 10 12:08:10.478759 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.478549 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2a3d73c-5524-4fb9-9e14-d51d6974db04" containerName="revoke-keys" Sep 10 12:08:10.478759 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.478563 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerName="istio-proxy" Sep 10 12:08:10.478759 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.478572 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcba9edd-88c8-4552-b985-c9c7bdb657ed" containerName="istio-proxy" Sep 10 12:08:10.480566 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.480549 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.483707 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.483689 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-316319-openshift-default-dockercfg-d6brs\"" Sep 10 12:08:10.497682 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.497658 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v"] Sep 10 12:08:10.650776 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.650735 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-credential-socket\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.650942 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.650780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f911bf6-0c4c-4787-95fd-758415991c41-istiod-ca-cert\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.650942 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.650807 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mglb5\" (UniqueName: \"kubernetes.io/projected/5f911bf6-0c4c-4787-95fd-758415991c41-kube-api-access-mglb5\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.650942 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.650884 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-istio-envoy\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.650942 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.650913 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f911bf6-0c4c-4787-95fd-758415991c41-istio-podinfo\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.650942 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.650938 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-workload-certs\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.651178 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.650959 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-workload-socket\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.651178 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.651055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f911bf6-0c4c-4787-95fd-758415991c41-istio-token\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.651178 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.651106 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-istio-data\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.752272 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.752195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f911bf6-0c4c-4787-95fd-758415991c41-istio-token\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.752272 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.752237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-istio-data\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.752272 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.752260 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-credential-socket\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.752550 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.752278 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f911bf6-0c4c-4787-95fd-758415991c41-istiod-ca-cert\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.752550 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.752460 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mglb5\" (UniqueName: \"kubernetes.io/projected/5f911bf6-0c4c-4787-95fd-758415991c41-kube-api-access-mglb5\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.752550 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.752527 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-istio-envoy\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.752705 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.752555 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f911bf6-0c4c-4787-95fd-758415991c41-istio-podinfo\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.752705 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.752592 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-workload-certs\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.752705 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.752606 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-istio-data\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.752705 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.752643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-workload-socket\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.752705 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.752652 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-credential-socket\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.752958 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.752939 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-workload-socket\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.753014 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.752970 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-workload-certs\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.753081 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.753058 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f911bf6-0c4c-4787-95fd-758415991c41-istiod-ca-cert\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.754985 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.754961 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-istio-envoy\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.755160 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.755128 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f911bf6-0c4c-4787-95fd-758415991c41-istio-podinfo\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.762115 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.762092 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f911bf6-0c4c-4787-95fd-758415991c41-istio-token\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.762250 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.762173 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mglb5\" (UniqueName: \"kubernetes.io/projected/5f911bf6-0c4c-4787-95fd-758415991c41-kube-api-access-mglb5\") pod \"e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:10.791131 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:10.791107 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:11.068159 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:11.068112 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v"] Sep 10 12:08:11.070365 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:08:11.070335 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f911bf6_0c4c_4787_95fd_758415991c41.slice/crio-cb70148bb437c6a1d7c7899f1ecd0c2d12970f7ab6813bf0490354cd9e973e58 WatchSource:0}: Error finding container cb70148bb437c6a1d7c7899f1ecd0c2d12970f7ab6813bf0490354cd9e973e58: Status 404 returned error can't find the container with id cb70148bb437c6a1d7c7899f1ecd0c2d12970f7ab6813bf0490354cd9e973e58 Sep 10 12:08:11.072602 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:11.072566 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:08:11.072689 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:11.072635 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:08:11.072689 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:11.072662 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:08:11.577817 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:11.577780 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" event={"ID":"5f911bf6-0c4c-4787-95fd-758415991c41","Type":"ContainerStarted","Data":"d490187727b7ee7153e513ce90ac72dabca18af47e330dbaee6fc86b64fa1489"} Sep 10 12:08:11.578217 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:11.577837 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" event={"ID":"5f911bf6-0c4c-4787-95fd-758415991c41","Type":"ContainerStarted","Data":"cb70148bb437c6a1d7c7899f1ecd0c2d12970f7ab6813bf0490354cd9e973e58"} Sep 10 12:08:11.598478 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:11.598423 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" podStartSLOduration=1.598403978 podStartE2EDuration="1.598403978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:08:11.597036735 +0000 UTC m=+999.476426167" watchObservedRunningTime="2026-09-10 12:08:11.598403978 +0000 UTC m=+999.477793414" Sep 10 12:08:11.791799 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:11.791768 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:12.796585 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:12.796557 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:13.069375 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:13.069294 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:08:13.589712 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:13.589681 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:13.590631 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:13.590610 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:29.053976 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:29.053927 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:08:29.631294 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:29.631253 2582 patch_prober.go:28] interesting pod/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 12:08:29.631538 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:29.631317 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 12:08:31.692258 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:31.692226 2582 generic.go:356] "Generic (PLEG): container finished" podID="5f911bf6-0c4c-4787-95fd-758415991c41" containerID="d490187727b7ee7153e513ce90ac72dabca18af47e330dbaee6fc86b64fa1489" exitCode=0 Sep 10 12:08:31.692680 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:31.692298 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" event={"ID":"5f911bf6-0c4c-4787-95fd-758415991c41","Type":"ContainerDied","Data":"d490187727b7ee7153e513ce90ac72dabca18af47e330dbaee6fc86b64fa1489"} Sep 10 12:08:31.692741 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:31.692725 2582 scope.go:117] "RemoveContainer" containerID="d490187727b7ee7153e513ce90ac72dabca18af47e330dbaee6fc86b64fa1489" Sep 10 12:08:31.693773 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:31.693747 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:08:31.693849 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:31.693819 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:08:31.693892 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:31.693851 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:08:32.701900 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:32.701855 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" event={"ID":"5f911bf6-0c4c-4787-95fd-758415991c41","Type":"ContainerStarted","Data":"a726561b48f41b6fdcbd058955de488337df01d2938933d30a95148e62da9281"} Sep 10 12:08:32.791716 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:32.791686 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:32.791861 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:32.791823 2582 patch_prober.go:28] interesting pod/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 10 12:08:32.791907 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:32.791878 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 10 12:08:33.791859 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:33.791824 2582 patch_prober.go:28] interesting pod/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 10 12:08:33.792267 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:33.791891 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 10 12:08:34.791619 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:34.791583 2582 patch_prober.go:28] interesting pod/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 10 12:08:34.791795 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:34.791645 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 10 12:08:35.791652 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:35.791624 2582 patch_prober.go:28] interesting pod/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 10 12:08:35.792023 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:35.791678 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 10 12:08:36.791855 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:36.791820 2582 patch_prober.go:28] interesting pod/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 10 12:08:36.792270 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:36.791872 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 10 12:08:37.791912 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:37.791879 2582 patch_prober.go:28] interesting pod/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 10 12:08:37.792331 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:37.791950 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 10 12:08:38.796111 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:38.796078 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:38.796612 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:38.796502 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:38.797334 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:38.797308 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:08:47.356400 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:47.356363 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:08:56.278271 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:56.278235 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z"] Sep 10 12:08:56.281192 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:56.281175 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z" Sep 10 12:08:56.283982 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:56.283958 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:08:56.284107 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:56.283991 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:08:56.292088 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:56.292065 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z"] Sep 10 12:08:56.403991 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:56.403938 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5632b970-057b-48a0-9983-886bb57603e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-316319-8kh6z\" (UID: \"5632b970-057b-48a0-9983-886bb57603e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z" Sep 10 12:08:56.504982 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:56.504940 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5632b970-057b-48a0-9983-886bb57603e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-316319-8kh6z\" (UID: \"5632b970-057b-48a0-9983-886bb57603e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z" Sep 10 12:08:56.505837 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:56.505804 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5632b970-057b-48a0-9983-886bb57603e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-316319-8kh6z\" (UID: \"5632b970-057b-48a0-9983-886bb57603e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z" Sep 10 12:08:56.592319 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:56.592221 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z" Sep 10 12:08:56.889428 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:56.889404 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z"] Sep 10 12:08:56.892089 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:08:56.892061 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5632b970_057b_48a0_9983_886bb57603e9.slice/crio-ea2c0c7eec2a9b37244ef163f729b91a4d9b1f411bad792b25431d4ceb6734dc WatchSource:0}: Error finding container ea2c0c7eec2a9b37244ef163f729b91a4d9b1f411bad792b25431d4ceb6734dc: Status 404 returned error can't find the container with id ea2c0c7eec2a9b37244ef163f729b91a4d9b1f411bad792b25431d4ceb6734dc Sep 10 12:08:57.375787 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.375742 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49"] Sep 10 12:08:57.378357 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.378334 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.381134 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.381113 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0f6353c1-openshift-default-dockercfg-6jlbw\"" Sep 10 12:08:57.407006 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.406975 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49"] Sep 10 12:08:57.514432 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.514338 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f5472cb-8a14-47be-a9aa-a733c366169e-istiod-ca-cert\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.514432 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.514399 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4d8t\" (UniqueName: \"kubernetes.io/projected/2f5472cb-8a14-47be-a9aa-a733c366169e-kube-api-access-v4d8t\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.514432 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.514432 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-workload-certs\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.514714 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.514522 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-envoy\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.514714 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.514596 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-workload-socket\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.514714 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.514648 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-credential-socket\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.514714 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.514703 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-token\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.514877 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.514732 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-data\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.514877 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.514781 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-podinfo\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.615261 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.615229 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4d8t\" (UniqueName: \"kubernetes.io/projected/2f5472cb-8a14-47be-a9aa-a733c366169e-kube-api-access-v4d8t\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.615261 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.615266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-workload-certs\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.615498 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.615299 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-envoy\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.615498 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.615353 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-workload-socket\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.615498 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.615399 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-credential-socket\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.615656 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.615518 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-token\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.615656 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.615572 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-data\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.615656 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.615630 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-podinfo\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.615815 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.615689 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f5472cb-8a14-47be-a9aa-a733c366169e-istiod-ca-cert\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.615815 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.615737 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-workload-certs\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.615815 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.615802 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-workload-socket\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.615958 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.615853 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-credential-socket\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.616159 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.616091 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-data\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.616317 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.616229 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f5472cb-8a14-47be-a9aa-a733c366169e-istiod-ca-cert\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.617770 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.617752 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-envoy\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.618084 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.618067 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-podinfo\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.626375 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.626351 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4d8t\" (UniqueName: \"kubernetes.io/projected/2f5472cb-8a14-47be-a9aa-a733c366169e-kube-api-access-v4d8t\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.627400 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.627383 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-token\") pod \"e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.688161 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.688119 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:57.864811 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.864725 2582 generic.go:356] "Generic (PLEG): container finished" podID="5632b970-057b-48a0-9983-886bb57603e9" containerID="b25c0d28bb201f853b89d40ecff98b525a64c0ba16cbdced98064dc10a4466a6" exitCode=0 Sep 10 12:08:57.864939 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.864809 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z" event={"ID":"5632b970-057b-48a0-9983-886bb57603e9","Type":"ContainerDied","Data":"b25c0d28bb201f853b89d40ecff98b525a64c0ba16cbdced98064dc10a4466a6"} Sep 10 12:08:57.864939 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.864856 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z" event={"ID":"5632b970-057b-48a0-9983-886bb57603e9","Type":"ContainerStarted","Data":"ea2c0c7eec2a9b37244ef163f729b91a4d9b1f411bad792b25431d4ceb6734dc"} Sep 10 12:08:57.965855 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.965828 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49"] Sep 10 12:08:57.967317 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:08:57.967278 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5472cb_8a14_47be_a9aa_a733c366169e.slice/crio-17905f4412fa2ad67728e04e642c77ffaa68894ed01717e7e1c1c91a22618121 WatchSource:0}: Error finding container 17905f4412fa2ad67728e04e642c77ffaa68894ed01717e7e1c1c91a22618121: Status 404 returned error can't find the container with id 17905f4412fa2ad67728e04e642c77ffaa68894ed01717e7e1c1c91a22618121 Sep 10 12:08:57.969512 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.969479 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:08:57.969610 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.969541 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:08:57.969610 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:57.969567 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:08:58.872369 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:58.872334 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" event={"ID":"2f5472cb-8a14-47be-a9aa-a733c366169e","Type":"ContainerStarted","Data":"3f2de8be559dad4ac68232e19fa964f8e14e4e1aee2babef6aa79a400fdcfa46"} Sep 10 12:08:58.872794 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:58.872395 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" event={"ID":"2f5472cb-8a14-47be-a9aa-a733c366169e","Type":"ContainerStarted","Data":"17905f4412fa2ad67728e04e642c77ffaa68894ed01717e7e1c1c91a22618121"} Sep 10 12:08:58.896019 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:58.895959 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" podStartSLOduration=1.895939137 podStartE2EDuration="1.895939137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:08:58.893746137 +0000 UTC m=+1046.773135568" watchObservedRunningTime="2026-09-10 12:08:58.895939137 +0000 UTC m=+1046.775328570" Sep 10 12:08:59.018888 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:59.018858 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z" Sep 10 12:08:59.028243 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:59.028223 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5632b970-057b-48a0-9983-886bb57603e9-maas-api-service-ca\") pod \"5632b970-057b-48a0-9983-886bb57603e9\" (UID: \"5632b970-057b-48a0-9983-886bb57603e9\") " Sep 10 12:08:59.028543 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:59.028522 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5632b970-057b-48a0-9983-886bb57603e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5632b970-057b-48a0-9983-886bb57603e9" (UID: "5632b970-057b-48a0-9983-886bb57603e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:08:59.129514 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:59.129430 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5632b970-057b-48a0-9983-886bb57603e9-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:08:59.688610 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:59.688581 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:59.693882 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:59.693856 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:59.879025 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:59.878997 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z" Sep 10 12:08:59.879476 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:59.879031 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z" event={"ID":"5632b970-057b-48a0-9983-886bb57603e9","Type":"ContainerDied","Data":"ea2c0c7eec2a9b37244ef163f729b91a4d9b1f411bad792b25431d4ceb6734dc"} Sep 10 12:08:59.879476 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:59.879071 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea2c0c7eec2a9b37244ef163f729b91a4d9b1f411bad792b25431d4ceb6734dc" Sep 10 12:08:59.879476 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:59.879377 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:08:59.880254 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:08:59.880238 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:09:06.459188 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:06.459153 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:09:07.330815 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:07.330779 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z"] Sep 10 12:09:07.335801 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:07.335775 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-316319-8kh6z"] Sep 10 12:09:08.632691 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:08.632656 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5632b970-057b-48a0-9983-886bb57603e9" path="/var/lib/kubelet/pods/5632b970-057b-48a0-9983-886bb57603e9/volumes" Sep 10 12:09:15.920335 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:15.920294 2582 patch_prober.go:28] interesting pod/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 12:09:15.920741 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:15.920370 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 12:09:16.920950 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.920875 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q"] Sep 10 12:09:16.921355 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.921298 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5632b970-057b-48a0-9983-886bb57603e9" containerName="revoke-keys" Sep 10 12:09:16.921355 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.921310 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5632b970-057b-48a0-9983-886bb57603e9" containerName="revoke-keys" Sep 10 12:09:16.921427 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.921379 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5632b970-057b-48a0-9983-886bb57603e9" containerName="revoke-keys" Sep 10 12:09:16.923445 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.923430 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q" Sep 10 12:09:16.926566 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.926544 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:09:16.926701 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.926690 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:09:16.937753 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.937727 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q"] Sep 10 12:09:16.982833 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.982802 2582 generic.go:356] "Generic (PLEG): container finished" podID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerID="3f2de8be559dad4ac68232e19fa964f8e14e4e1aee2babef6aa79a400fdcfa46" exitCode=0 Sep 10 12:09:16.982976 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.982875 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" event={"ID":"2f5472cb-8a14-47be-a9aa-a733c366169e","Type":"ContainerDied","Data":"3f2de8be559dad4ac68232e19fa964f8e14e4e1aee2babef6aa79a400fdcfa46"} Sep 10 12:09:16.983293 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.983280 2582 scope.go:117] "RemoveContainer" containerID="3f2de8be559dad4ac68232e19fa964f8e14e4e1aee2babef6aa79a400fdcfa46" Sep 10 12:09:16.984335 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.984309 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:09:16.984404 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.984376 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:09:16.984441 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:16.984412 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:09:17.014200 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:17.014128 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c374ba13-7842-4b7f-bfb2-1f79bcaebf4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89d91a-g668q\" (UID: \"c374ba13-7842-4b7f-bfb2-1f79bcaebf4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q" Sep 10 12:09:17.115376 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:17.115343 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c374ba13-7842-4b7f-bfb2-1f79bcaebf4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89d91a-g668q\" (UID: \"c374ba13-7842-4b7f-bfb2-1f79bcaebf4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q" Sep 10 12:09:17.116261 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:17.116238 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c374ba13-7842-4b7f-bfb2-1f79bcaebf4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89d91a-g668q\" (UID: \"c374ba13-7842-4b7f-bfb2-1f79bcaebf4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q" Sep 10 12:09:17.233514 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:17.233481 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q" Sep 10 12:09:17.510887 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:17.510861 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q"] Sep 10 12:09:17.513265 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:09:17.513237 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc374ba13_7842_4b7f_bfb2_1f79bcaebf4c.slice/crio-8d7be78241196a0c0e3f5171150e107494734cac92f08f26b9706114d414923c WatchSource:0}: Error finding container 8d7be78241196a0c0e3f5171150e107494734cac92f08f26b9706114d414923c: Status 404 returned error can't find the container with id 8d7be78241196a0c0e3f5171150e107494734cac92f08f26b9706114d414923c Sep 10 12:09:17.996814 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:17.996770 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q" event={"ID":"c374ba13-7842-4b7f-bfb2-1f79bcaebf4c","Type":"ContainerStarted","Data":"914bdfa44f4371e193f9dd404411774d4d3c4910bb09e80a747c9bc2cdcb301b"} Sep 10 12:09:17.997311 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:17.996884 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q" event={"ID":"c374ba13-7842-4b7f-bfb2-1f79bcaebf4c","Type":"ContainerStarted","Data":"8d7be78241196a0c0e3f5171150e107494734cac92f08f26b9706114d414923c"} Sep 10 12:09:18.005057 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:18.005026 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" event={"ID":"2f5472cb-8a14-47be-a9aa-a733c366169e","Type":"ContainerStarted","Data":"e36c2fe733c2ab2859717ed1646b97fa4a35c5c7dabd876738bc0eb12f52e53f"} Sep 10 12:09:18.020376 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:18.020312 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q" podStartSLOduration=2.020292271 podStartE2EDuration="2.020292271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:09:18.019579312 +0000 UTC m=+1065.898968754" watchObservedRunningTime="2026-09-10 12:09:18.020292271 +0000 UTC m=+1065.899681704" Sep 10 12:09:18.689114 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:18.689082 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:09:18.689338 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:18.689319 2582 patch_prober.go:28] interesting pod/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 12:09:18.689380 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:18.689364 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 12:09:19.017481 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:19.017387 2582 generic.go:356] "Generic (PLEG): container finished" podID="c374ba13-7842-4b7f-bfb2-1f79bcaebf4c" containerID="914bdfa44f4371e193f9dd404411774d4d3c4910bb09e80a747c9bc2cdcb301b" exitCode=0 Sep 10 12:09:19.018882 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:19.018851 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q" event={"ID":"c374ba13-7842-4b7f-bfb2-1f79bcaebf4c","Type":"ContainerDied","Data":"914bdfa44f4371e193f9dd404411774d4d3c4910bb09e80a747c9bc2cdcb301b"} Sep 10 12:09:19.320114 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:19.320022 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v"] Sep 10 12:09:19.321517 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:19.321486 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" containerID="cri-o://a726561b48f41b6fdcbd058955de488337df01d2938933d30a95148e62da9281" gracePeriod=30 Sep 10 12:09:19.688711 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:19.688681 2582 patch_prober.go:28] interesting pod/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 12:09:19.688876 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:19.688737 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 12:09:20.055137 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.054913 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5"] Sep 10 12:09:20.058654 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.058630 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.061835 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.061811 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-c8a24e-openshift-default-dockercfg-rwml7\"" Sep 10 12:09:20.071250 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.071226 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5"] Sep 10 12:09:20.150015 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.149898 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-workload-certs\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.150015 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.149967 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-istio-data\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.151065 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.150660 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851f1d92-74cc-456f-a059-913d2368e85d-istio-token\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.151065 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.150744 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851f1d92-74cc-456f-a059-913d2368e85d-istio-podinfo\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.151065 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.150778 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-workload-socket\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.151065 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.150828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-credential-socket\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.151065 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.150867 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-istio-envoy\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.151065 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.150897 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxtb\" (UniqueName: \"kubernetes.io/projected/851f1d92-74cc-456f-a059-913d2368e85d-kube-api-access-tjxtb\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.151065 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.150935 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851f1d92-74cc-456f-a059-913d2368e85d-istiod-ca-cert\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.243691 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.243669 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q" Sep 10 12:09:20.251773 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.251750 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-istio-envoy\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.251865 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.251784 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxtb\" (UniqueName: \"kubernetes.io/projected/851f1d92-74cc-456f-a059-913d2368e85d-kube-api-access-tjxtb\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.251865 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.251808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851f1d92-74cc-456f-a059-913d2368e85d-istiod-ca-cert\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.251865 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.251846 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-workload-certs\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.251865 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.251861 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-istio-data\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.252072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.251902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851f1d92-74cc-456f-a059-913d2368e85d-istio-token\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.252072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.251937 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851f1d92-74cc-456f-a059-913d2368e85d-istio-podinfo\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.252072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.251958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-workload-socket\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.252072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.251991 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-credential-socket\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.252575 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.252551 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-workload-certs\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.252711 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.252596 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-istio-data\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.252711 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.252632 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-credential-socket\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.252711 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.252660 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-workload-socket\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.253049 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.252859 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851f1d92-74cc-456f-a059-913d2368e85d-istiod-ca-cert\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.254719 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.254699 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-istio-envoy\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.254719 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.254712 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851f1d92-74cc-456f-a059-913d2368e85d-istio-podinfo\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.263340 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.263298 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851f1d92-74cc-456f-a059-913d2368e85d-istio-token\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.266290 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.266244 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxtb\" (UniqueName: \"kubernetes.io/projected/851f1d92-74cc-456f-a059-913d2368e85d-kube-api-access-tjxtb\") pod \"e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.353556 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.353466 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c374ba13-7842-4b7f-bfb2-1f79bcaebf4c-maas-api-service-ca\") pod \"c374ba13-7842-4b7f-bfb2-1f79bcaebf4c\" (UID: \"c374ba13-7842-4b7f-bfb2-1f79bcaebf4c\") " Sep 10 12:09:20.353848 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.353822 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c374ba13-7842-4b7f-bfb2-1f79bcaebf4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c374ba13-7842-4b7f-bfb2-1f79bcaebf4c" (UID: "c374ba13-7842-4b7f-bfb2-1f79bcaebf4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:09:20.371436 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.371411 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:20.454671 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.454639 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c374ba13-7842-4b7f-bfb2-1f79bcaebf4c-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:20.674695 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.674672 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5"] Sep 10 12:09:20.676467 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:09:20.676440 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851f1d92_74cc_456f_a059_913d2368e85d.slice/crio-324a3aac1a1568b2fdedf028aafd078eb3e18068802dde8ad90de85059535752 WatchSource:0}: Error finding container 324a3aac1a1568b2fdedf028aafd078eb3e18068802dde8ad90de85059535752: Status 404 returned error can't find the container with id 324a3aac1a1568b2fdedf028aafd078eb3e18068802dde8ad90de85059535752 Sep 10 12:09:20.678732 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.678698 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:09:20.678808 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.678763 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:09:20.678842 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.678810 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:09:20.688696 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.688672 2582 patch_prober.go:28] interesting pod/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 12:09:20.688782 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:20.688709 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 12:09:21.037492 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:21.037453 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q" event={"ID":"c374ba13-7842-4b7f-bfb2-1f79bcaebf4c","Type":"ContainerDied","Data":"8d7be78241196a0c0e3f5171150e107494734cac92f08f26b9706114d414923c"} Sep 10 12:09:21.037742 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:21.037696 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d7be78241196a0c0e3f5171150e107494734cac92f08f26b9706114d414923c" Sep 10 12:09:21.037859 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:21.037476 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q" Sep 10 12:09:21.039541 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:21.039378 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" event={"ID":"851f1d92-74cc-456f-a059-913d2368e85d","Type":"ContainerStarted","Data":"9d75bfa24b59187bf802d22b1dcc7b07b777e5f9ad2c8ea643145fce877e347d"} Sep 10 12:09:21.039541 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:21.039426 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" event={"ID":"851f1d92-74cc-456f-a059-913d2368e85d","Type":"ContainerStarted","Data":"324a3aac1a1568b2fdedf028aafd078eb3e18068802dde8ad90de85059535752"} Sep 10 12:09:21.061844 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:21.061787 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" podStartSLOduration=1.061772926 podStartE2EDuration="1.061772926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:09:21.059912472 +0000 UTC m=+1068.939301911" watchObservedRunningTime="2026-09-10 12:09:21.061772926 +0000 UTC m=+1068.941162351" Sep 10 12:09:21.371897 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:21.371804 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:21.373360 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:21.373335 2582 patch_prober.go:28] interesting pod/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 12:09:21.373523 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:21.373386 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" podUID="851f1d92-74cc-456f-a059-913d2368e85d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 12:09:21.689451 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:21.689416 2582 patch_prober.go:28] interesting pod/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 12:09:21.689649 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:21.689479 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 12:09:22.376368 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:22.376337 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:22.688605 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:22.688575 2582 patch_prober.go:28] interesting pod/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 10 12:09:22.688768 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:22.688633 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 10 12:09:23.051343 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:23.051253 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:23.052174 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:23.052128 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:09:23.800014 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:23.799916 2582 patch_prober.go:28] interesting pod/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 12:09:23.800014 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:23.799998 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 12:09:23.817211 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:23.817138 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:09:23.817683 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:23.817663 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:09:23.818418 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:23.818401 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:09:24.574737 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.574712 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:09:24.699893 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.699865 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f911bf6-0c4c-4787-95fd-758415991c41-istio-podinfo\") pod \"5f911bf6-0c4c-4787-95fd-758415991c41\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " Sep 10 12:09:24.700053 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.699907 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f911bf6-0c4c-4787-95fd-758415991c41-istiod-ca-cert\") pod \"5f911bf6-0c4c-4787-95fd-758415991c41\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " Sep 10 12:09:24.700053 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.699951 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-workload-socket\") pod \"5f911bf6-0c4c-4787-95fd-758415991c41\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " Sep 10 12:09:24.700053 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.699984 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-istio-data\") pod \"5f911bf6-0c4c-4787-95fd-758415991c41\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " Sep 10 12:09:24.700053 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.700047 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f911bf6-0c4c-4787-95fd-758415991c41-istio-token\") pod \"5f911bf6-0c4c-4787-95fd-758415991c41\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " Sep 10 12:09:24.700274 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.700099 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-istio-envoy\") pod \"5f911bf6-0c4c-4787-95fd-758415991c41\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " Sep 10 12:09:24.700274 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.700138 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-credential-socket\") pod \"5f911bf6-0c4c-4787-95fd-758415991c41\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " Sep 10 12:09:24.700274 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.700195 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-workload-certs\") pod \"5f911bf6-0c4c-4787-95fd-758415991c41\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " Sep 10 12:09:24.700274 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.700248 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mglb5\" (UniqueName: \"kubernetes.io/projected/5f911bf6-0c4c-4787-95fd-758415991c41-kube-api-access-mglb5\") pod \"5f911bf6-0c4c-4787-95fd-758415991c41\" (UID: \"5f911bf6-0c4c-4787-95fd-758415991c41\") " Sep 10 12:09:24.700556 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.700346 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f911bf6-0c4c-4787-95fd-758415991c41" (UID: "5f911bf6-0c4c-4787-95fd-758415991c41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:09:24.700556 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.700416 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f911bf6-0c4c-4787-95fd-758415991c41-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f911bf6-0c4c-4787-95fd-758415991c41" (UID: "5f911bf6-0c4c-4787-95fd-758415991c41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:09:24.700556 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.700484 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f911bf6-0c4c-4787-95fd-758415991c41" (UID: "5f911bf6-0c4c-4787-95fd-758415991c41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:09:24.700556 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.700521 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f911bf6-0c4c-4787-95fd-758415991c41" (UID: "5f911bf6-0c4c-4787-95fd-758415991c41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:09:24.700939 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.700915 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:24.701063 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.700943 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:24.701063 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.700959 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:24.701063 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.700974 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f911bf6-0c4c-4787-95fd-758415991c41-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:24.701270 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.701217 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f911bf6-0c4c-4787-95fd-758415991c41" (UID: "5f911bf6-0c4c-4787-95fd-758415991c41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:09:24.702566 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.702535 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f911bf6-0c4c-4787-95fd-758415991c41-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f911bf6-0c4c-4787-95fd-758415991c41" (UID: "5f911bf6-0c4c-4787-95fd-758415991c41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:09:24.702855 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.702823 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f911bf6-0c4c-4787-95fd-758415991c41-kube-api-access-mglb5" (OuterVolumeSpecName: "kube-api-access-mglb5") pod "5f911bf6-0c4c-4787-95fd-758415991c41" (UID: "5f911bf6-0c4c-4787-95fd-758415991c41"). InnerVolumeSpecName "kube-api-access-mglb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:09:24.703045 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.703018 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f911bf6-0c4c-4787-95fd-758415991c41-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f911bf6-0c4c-4787-95fd-758415991c41" (UID: "5f911bf6-0c4c-4787-95fd-758415991c41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:09:24.703159 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.703102 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f911bf6-0c4c-4787-95fd-758415991c41" (UID: "5f911bf6-0c4c-4787-95fd-758415991c41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:09:24.801993 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.801960 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mglb5\" (UniqueName: \"kubernetes.io/projected/5f911bf6-0c4c-4787-95fd-758415991c41-kube-api-access-mglb5\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:24.801993 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.801988 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f911bf6-0c4c-4787-95fd-758415991c41-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:24.801993 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.801997 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f911bf6-0c4c-4787-95fd-758415991c41-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:24.802448 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.802005 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:24.802448 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:24.802013 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f911bf6-0c4c-4787-95fd-758415991c41-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:25.063675 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:25.063589 2582 generic.go:356] "Generic (PLEG): container finished" podID="5f911bf6-0c4c-4787-95fd-758415991c41" containerID="a726561b48f41b6fdcbd058955de488337df01d2938933d30a95148e62da9281" exitCode=0 Sep 10 12:09:25.063675 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:25.063638 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" Sep 10 12:09:25.063852 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:25.063678 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" event={"ID":"5f911bf6-0c4c-4787-95fd-758415991c41","Type":"ContainerDied","Data":"a726561b48f41b6fdcbd058955de488337df01d2938933d30a95148e62da9281"} Sep 10 12:09:25.063852 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:25.063734 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v" event={"ID":"5f911bf6-0c4c-4787-95fd-758415991c41","Type":"ContainerDied","Data":"cb70148bb437c6a1d7c7899f1ecd0c2d12970f7ab6813bf0490354cd9e973e58"} Sep 10 12:09:25.063852 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:25.063759 2582 scope.go:117] "RemoveContainer" containerID="a726561b48f41b6fdcbd058955de488337df01d2938933d30a95148e62da9281" Sep 10 12:09:25.073935 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:25.073916 2582 scope.go:117] "RemoveContainer" containerID="d490187727b7ee7153e513ce90ac72dabca18af47e330dbaee6fc86b64fa1489" Sep 10 12:09:25.082676 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:25.082655 2582 scope.go:117] "RemoveContainer" containerID="a726561b48f41b6fdcbd058955de488337df01d2938933d30a95148e62da9281" Sep 10 12:09:25.082989 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:09:25.082963 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a726561b48f41b6fdcbd058955de488337df01d2938933d30a95148e62da9281\": container with ID starting with a726561b48f41b6fdcbd058955de488337df01d2938933d30a95148e62da9281 not found: ID does not exist" containerID="a726561b48f41b6fdcbd058955de488337df01d2938933d30a95148e62da9281" Sep 10 12:09:25.083036 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:25.083002 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a726561b48f41b6fdcbd058955de488337df01d2938933d30a95148e62da9281"} err="failed to get container status \"a726561b48f41b6fdcbd058955de488337df01d2938933d30a95148e62da9281\": rpc error: code = NotFound desc = could not find container \"a726561b48f41b6fdcbd058955de488337df01d2938933d30a95148e62da9281\": container with ID starting with a726561b48f41b6fdcbd058955de488337df01d2938933d30a95148e62da9281 not found: ID does not exist" Sep 10 12:09:25.083036 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:25.083028 2582 scope.go:117] "RemoveContainer" containerID="d490187727b7ee7153e513ce90ac72dabca18af47e330dbaee6fc86b64fa1489" Sep 10 12:09:25.083340 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:09:25.083319 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d490187727b7ee7153e513ce90ac72dabca18af47e330dbaee6fc86b64fa1489\": container with ID starting with d490187727b7ee7153e513ce90ac72dabca18af47e330dbaee6fc86b64fa1489 not found: ID does not exist" containerID="d490187727b7ee7153e513ce90ac72dabca18af47e330dbaee6fc86b64fa1489" Sep 10 12:09:25.083413 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:25.083355 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d490187727b7ee7153e513ce90ac72dabca18af47e330dbaee6fc86b64fa1489"} err="failed to get container status \"d490187727b7ee7153e513ce90ac72dabca18af47e330dbaee6fc86b64fa1489\": rpc error: code = NotFound desc = could not find container \"d490187727b7ee7153e513ce90ac72dabca18af47e330dbaee6fc86b64fa1489\": container with ID starting with d490187727b7ee7153e513ce90ac72dabca18af47e330dbaee6fc86b64fa1489 not found: ID does not exist" Sep 10 12:09:25.091621 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:25.091597 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v"] Sep 10 12:09:25.101035 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:25.101008 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-316319-openshift-default-79dc49b7d4-8sf2v"] Sep 10 12:09:26.630161 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:26.630111 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" path="/var/lib/kubelet/pods/5f911bf6-0c4c-4787-95fd-758415991c41/volumes" Sep 10 12:09:27.906707 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:27.906670 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q"] Sep 10 12:09:27.909537 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:27.909513 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89d91a-g668q"] Sep 10 12:09:28.553830 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:28.553797 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:09:28.629884 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:28.629845 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c374ba13-7842-4b7f-bfb2-1f79bcaebf4c" path="/var/lib/kubelet/pods/c374ba13-7842-4b7f-bfb2-1f79bcaebf4c/volumes" Sep 10 12:09:35.737781 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.737748 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f"] Sep 10 12:09:35.738205 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.738137 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" Sep 10 12:09:35.738205 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.738168 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" Sep 10 12:09:35.738205 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.738205 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c374ba13-7842-4b7f-bfb2-1f79bcaebf4c" containerName="revoke-keys" Sep 10 12:09:35.738320 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.738212 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c374ba13-7842-4b7f-bfb2-1f79bcaebf4c" containerName="revoke-keys" Sep 10 12:09:35.738320 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.738270 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" Sep 10 12:09:35.738320 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.738280 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c374ba13-7842-4b7f-bfb2-1f79bcaebf4c" containerName="revoke-keys" Sep 10 12:09:35.738320 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.738288 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" Sep 10 12:09:35.742566 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.742548 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f" Sep 10 12:09:35.747056 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.747034 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:09:35.747224 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.747063 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:09:35.753861 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.753839 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f"] Sep 10 12:09:35.811386 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.811358 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24fa3df3-a321-4677-96ac-97f1d6d60b6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f\" (UID: \"24fa3df3-a321-4677-96ac-97f1d6d60b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f" Sep 10 12:09:35.912496 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.912467 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24fa3df3-a321-4677-96ac-97f1d6d60b6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f\" (UID: \"24fa3df3-a321-4677-96ac-97f1d6d60b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f" Sep 10 12:09:35.913015 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:35.912996 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24fa3df3-a321-4677-96ac-97f1d6d60b6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f\" (UID: \"24fa3df3-a321-4677-96ac-97f1d6d60b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f" Sep 10 12:09:36.054096 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:36.054006 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f" Sep 10 12:09:36.352321 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:36.352296 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f"] Sep 10 12:09:36.355065 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:09:36.355034 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fa3df3_a321_4677_96ac_97f1d6d60b6f.slice/crio-d5448afab8086f31497026286e5d9bd2ec82713ae22a6f37699c65731f58f0a9 WatchSource:0}: Error finding container d5448afab8086f31497026286e5d9bd2ec82713ae22a6f37699c65731f58f0a9: Status 404 returned error can't find the container with id d5448afab8086f31497026286e5d9bd2ec82713ae22a6f37699c65731f58f0a9 Sep 10 12:09:37.141765 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:37.141726 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f" event={"ID":"24fa3df3-a321-4677-96ac-97f1d6d60b6f","Type":"ContainerStarted","Data":"2125c9c1988a1de3f55a0750e5cf39ed18133e68dfc717176fbf2af14b660ae6"} Sep 10 12:09:37.142250 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:37.141816 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f" event={"ID":"24fa3df3-a321-4677-96ac-97f1d6d60b6f","Type":"ContainerStarted","Data":"d5448afab8086f31497026286e5d9bd2ec82713ae22a6f37699c65731f58f0a9"} Sep 10 12:09:37.158402 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:37.158360 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f" podStartSLOduration=2.158345624 podStartE2EDuration="2.158345624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:09:37.156918249 +0000 UTC m=+1085.036307682" watchObservedRunningTime="2026-09-10 12:09:37.158345624 +0000 UTC m=+1085.037735057" Sep 10 12:09:37.756740 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:37.756703 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:09:38.150262 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:38.150136 2582 generic.go:356] "Generic (PLEG): container finished" podID="24fa3df3-a321-4677-96ac-97f1d6d60b6f" containerID="2125c9c1988a1de3f55a0750e5cf39ed18133e68dfc717176fbf2af14b660ae6" exitCode=0 Sep 10 12:09:38.150262 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:38.150247 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f" event={"ID":"24fa3df3-a321-4677-96ac-97f1d6d60b6f","Type":"ContainerDied","Data":"2125c9c1988a1de3f55a0750e5cf39ed18133e68dfc717176fbf2af14b660ae6"} Sep 10 12:09:39.294186 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:39.294135 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f" Sep 10 12:09:39.344531 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:39.344498 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24fa3df3-a321-4677-96ac-97f1d6d60b6f-maas-api-service-ca\") pod \"24fa3df3-a321-4677-96ac-97f1d6d60b6f\" (UID: \"24fa3df3-a321-4677-96ac-97f1d6d60b6f\") " Sep 10 12:09:39.344836 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:39.344802 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24fa3df3-a321-4677-96ac-97f1d6d60b6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24fa3df3-a321-4677-96ac-97f1d6d60b6f" (UID: "24fa3df3-a321-4677-96ac-97f1d6d60b6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:09:39.344915 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:39.344904 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24fa3df3-a321-4677-96ac-97f1d6d60b6f-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:40.163757 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:40.163720 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f" event={"ID":"24fa3df3-a321-4677-96ac-97f1d6d60b6f","Type":"ContainerDied","Data":"d5448afab8086f31497026286e5d9bd2ec82713ae22a6f37699c65731f58f0a9"} Sep 10 12:09:40.163934 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:40.163762 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f" Sep 10 12:09:40.163934 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:40.163778 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5448afab8086f31497026286e5d9bd2ec82713ae22a6f37699c65731f58f0a9" Sep 10 12:09:47.066807 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:47.066771 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f"] Sep 10 12:09:47.068901 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:47.068881 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-425dc7-58l7f"] Sep 10 12:09:48.630491 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:48.630460 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24fa3df3-a321-4677-96ac-97f1d6d60b6f" path="/var/lib/kubelet/pods/24fa3df3-a321-4677-96ac-97f1d6d60b6f/volumes" Sep 10 12:09:49.497596 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:49.497564 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:09:52.845634 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:52.845591 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49"] Sep 10 12:09:52.846112 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:52.845955 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" containerID="cri-o://e36c2fe733c2ab2859717ed1646b97fa4a35c5c7dabd876738bc0eb12f52e53f" gracePeriod=30 Sep 10 12:09:53.819780 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:53.819745 2582 patch_prober.go:28] interesting pod/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 12:09:53.819940 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:53.819818 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 12:09:55.939103 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:55.939071 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5"] Sep 10 12:09:55.939497 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:55.939320 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" podUID="851f1d92-74cc-456f-a059-913d2368e85d" containerName="istio-proxy" containerID="cri-o://9d75bfa24b59187bf802d22b1dcc7b07b777e5f9ad2c8ea643145fce877e347d" gracePeriod=30 Sep 10 12:09:56.140932 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:56.140896 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc"] Sep 10 12:09:56.141127 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:56.141101 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerName="istio-proxy" containerID="cri-o://1f6219b7814fed0acb2daddd08c8d3defa7b79d8c7366c7739ad0706c6f9e67a" gracePeriod=30 Sep 10 12:09:58.100068 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.100044 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:09:58.239653 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.239616 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-podinfo\") pod \"2f5472cb-8a14-47be-a9aa-a733c366169e\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " Sep 10 12:09:58.239653 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.239655 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f5472cb-8a14-47be-a9aa-a733c366169e-istiod-ca-cert\") pod \"2f5472cb-8a14-47be-a9aa-a733c366169e\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " Sep 10 12:09:58.239891 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.239677 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4d8t\" (UniqueName: \"kubernetes.io/projected/2f5472cb-8a14-47be-a9aa-a733c366169e-kube-api-access-v4d8t\") pod \"2f5472cb-8a14-47be-a9aa-a733c366169e\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " Sep 10 12:09:58.239891 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.239695 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-workload-socket\") pod \"2f5472cb-8a14-47be-a9aa-a733c366169e\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " Sep 10 12:09:58.239891 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.239724 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-credential-socket\") pod \"2f5472cb-8a14-47be-a9aa-a733c366169e\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " Sep 10 12:09:58.239891 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.239754 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-data\") pod \"2f5472cb-8a14-47be-a9aa-a733c366169e\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " Sep 10 12:09:58.239891 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.239795 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-token\") pod \"2f5472cb-8a14-47be-a9aa-a733c366169e\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " Sep 10 12:09:58.239891 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.239845 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-workload-certs\") pod \"2f5472cb-8a14-47be-a9aa-a733c366169e\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " Sep 10 12:09:58.239891 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.239876 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-envoy\") pod \"2f5472cb-8a14-47be-a9aa-a733c366169e\" (UID: \"2f5472cb-8a14-47be-a9aa-a733c366169e\") " Sep 10 12:09:58.240287 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.239982 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2f5472cb-8a14-47be-a9aa-a733c366169e" (UID: "2f5472cb-8a14-47be-a9aa-a733c366169e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:09:58.240287 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.240201 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5472cb-8a14-47be-a9aa-a733c366169e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2f5472cb-8a14-47be-a9aa-a733c366169e" (UID: "2f5472cb-8a14-47be-a9aa-a733c366169e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:09:58.240400 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.240310 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-data" (OuterVolumeSpecName: "istio-data") pod "2f5472cb-8a14-47be-a9aa-a733c366169e" (UID: "2f5472cb-8a14-47be-a9aa-a733c366169e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:09:58.240400 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.240326 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f5472cb-8a14-47be-a9aa-a733c366169e-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:58.240400 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.240362 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:58.240529 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.240445 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2f5472cb-8a14-47be-a9aa-a733c366169e" (UID: "2f5472cb-8a14-47be-a9aa-a733c366169e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:09:58.240644 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.240599 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2f5472cb-8a14-47be-a9aa-a733c366169e" (UID: "2f5472cb-8a14-47be-a9aa-a733c366169e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:09:58.242123 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.242094 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-token" (OuterVolumeSpecName: "istio-token") pod "2f5472cb-8a14-47be-a9aa-a733c366169e" (UID: "2f5472cb-8a14-47be-a9aa-a733c366169e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:09:58.242123 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.242117 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5472cb-8a14-47be-a9aa-a733c366169e-kube-api-access-v4d8t" (OuterVolumeSpecName: "kube-api-access-v4d8t") pod "2f5472cb-8a14-47be-a9aa-a733c366169e" (UID: "2f5472cb-8a14-47be-a9aa-a733c366169e"). InnerVolumeSpecName "kube-api-access-v4d8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:09:58.242583 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.242556 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2f5472cb-8a14-47be-a9aa-a733c366169e" (UID: "2f5472cb-8a14-47be-a9aa-a733c366169e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:09:58.242583 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.242564 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2f5472cb-8a14-47be-a9aa-a733c366169e" (UID: "2f5472cb-8a14-47be-a9aa-a733c366169e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:09:58.265799 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.265770 2582 generic.go:356] "Generic (PLEG): container finished" podID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerID="e36c2fe733c2ab2859717ed1646b97fa4a35c5c7dabd876738bc0eb12f52e53f" exitCode=0 Sep 10 12:09:58.265907 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.265870 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" Sep 10 12:09:58.265964 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.265861 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" event={"ID":"2f5472cb-8a14-47be-a9aa-a733c366169e","Type":"ContainerDied","Data":"e36c2fe733c2ab2859717ed1646b97fa4a35c5c7dabd876738bc0eb12f52e53f"} Sep 10 12:09:58.266009 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.265995 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49" event={"ID":"2f5472cb-8a14-47be-a9aa-a733c366169e","Type":"ContainerDied","Data":"17905f4412fa2ad67728e04e642c77ffaa68894ed01717e7e1c1c91a22618121"} Sep 10 12:09:58.266044 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.266027 2582 scope.go:117] "RemoveContainer" containerID="e36c2fe733c2ab2859717ed1646b97fa4a35c5c7dabd876738bc0eb12f52e53f" Sep 10 12:09:58.281551 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.281525 2582 scope.go:117] "RemoveContainer" containerID="3f2de8be559dad4ac68232e19fa964f8e14e4e1aee2babef6aa79a400fdcfa46" Sep 10 12:09:58.290005 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.289987 2582 scope.go:117] "RemoveContainer" containerID="e36c2fe733c2ab2859717ed1646b97fa4a35c5c7dabd876738bc0eb12f52e53f" Sep 10 12:09:58.290315 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:09:58.290293 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36c2fe733c2ab2859717ed1646b97fa4a35c5c7dabd876738bc0eb12f52e53f\": container with ID starting with e36c2fe733c2ab2859717ed1646b97fa4a35c5c7dabd876738bc0eb12f52e53f not found: ID does not exist" containerID="e36c2fe733c2ab2859717ed1646b97fa4a35c5c7dabd876738bc0eb12f52e53f" Sep 10 12:09:58.290375 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.290317 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36c2fe733c2ab2859717ed1646b97fa4a35c5c7dabd876738bc0eb12f52e53f"} err="failed to get container status \"e36c2fe733c2ab2859717ed1646b97fa4a35c5c7dabd876738bc0eb12f52e53f\": rpc error: code = NotFound desc = could not find container \"e36c2fe733c2ab2859717ed1646b97fa4a35c5c7dabd876738bc0eb12f52e53f\": container with ID starting with e36c2fe733c2ab2859717ed1646b97fa4a35c5c7dabd876738bc0eb12f52e53f not found: ID does not exist" Sep 10 12:09:58.290375 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.290332 2582 scope.go:117] "RemoveContainer" containerID="3f2de8be559dad4ac68232e19fa964f8e14e4e1aee2babef6aa79a400fdcfa46" Sep 10 12:09:58.290577 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:09:58.290561 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2de8be559dad4ac68232e19fa964f8e14e4e1aee2babef6aa79a400fdcfa46\": container with ID starting with 3f2de8be559dad4ac68232e19fa964f8e14e4e1aee2babef6aa79a400fdcfa46 not found: ID does not exist" containerID="3f2de8be559dad4ac68232e19fa964f8e14e4e1aee2babef6aa79a400fdcfa46" Sep 10 12:09:58.290625 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.290580 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2de8be559dad4ac68232e19fa964f8e14e4e1aee2babef6aa79a400fdcfa46"} err="failed to get container status \"3f2de8be559dad4ac68232e19fa964f8e14e4e1aee2babef6aa79a400fdcfa46\": rpc error: code = NotFound desc = could not find container \"3f2de8be559dad4ac68232e19fa964f8e14e4e1aee2babef6aa79a400fdcfa46\": container with ID starting with 3f2de8be559dad4ac68232e19fa964f8e14e4e1aee2babef6aa79a400fdcfa46 not found: ID does not exist" Sep 10 12:09:58.295292 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.295273 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49"] Sep 10 12:09:58.299702 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.299679 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0f6353c1-openshift-default-796bf8fc96-xtg49"] Sep 10 12:09:58.341609 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.341584 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:58.341609 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.341607 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4d8t\" (UniqueName: \"kubernetes.io/projected/2f5472cb-8a14-47be-a9aa-a733c366169e-kube-api-access-v4d8t\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:58.341770 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.341624 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:58.341770 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.341633 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:58.341770 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.341641 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:58.341770 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.341656 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:58.341770 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.341674 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f5472cb-8a14-47be-a9aa-a733c366169e-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:09:58.631666 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:09:58.631580 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" path="/var/lib/kubelet/pods/2f5472cb-8a14-47be-a9aa-a733c366169e/volumes" Sep 10 12:10:01.266851 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.266824 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:10:01.288858 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.288830 2582 generic.go:356] "Generic (PLEG): container finished" podID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerID="1f6219b7814fed0acb2daddd08c8d3defa7b79d8c7366c7739ad0706c6f9e67a" exitCode=0 Sep 10 12:10:01.289025 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.288910 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" event={"ID":"db3c479c-8910-473c-a499-8b9e8d8d10b6","Type":"ContainerDied","Data":"1f6219b7814fed0acb2daddd08c8d3defa7b79d8c7366c7739ad0706c6f9e67a"} Sep 10 12:10:01.289025 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.288991 2582 scope.go:117] "RemoveContainer" containerID="8b0eb934823c024958fcdd1566d4473a95aca55a0331e01f3aba07db61018e82" Sep 10 12:10:01.290449 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.290426 2582 generic.go:356] "Generic (PLEG): container finished" podID="851f1d92-74cc-456f-a059-913d2368e85d" containerID="9d75bfa24b59187bf802d22b1dcc7b07b777e5f9ad2c8ea643145fce877e347d" exitCode=0 Sep 10 12:10:01.290568 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.290483 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" Sep 10 12:10:01.290568 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.290467 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" event={"ID":"851f1d92-74cc-456f-a059-913d2368e85d","Type":"ContainerDied","Data":"9d75bfa24b59187bf802d22b1dcc7b07b777e5f9ad2c8ea643145fce877e347d"} Sep 10 12:10:01.290655 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.290616 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5" event={"ID":"851f1d92-74cc-456f-a059-913d2368e85d","Type":"ContainerDied","Data":"324a3aac1a1568b2fdedf028aafd078eb3e18068802dde8ad90de85059535752"} Sep 10 12:10:01.302377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.302342 2582 scope.go:117] "RemoveContainer" containerID="9d75bfa24b59187bf802d22b1dcc7b07b777e5f9ad2c8ea643145fce877e347d" Sep 10 12:10:01.317068 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.317046 2582 scope.go:117] "RemoveContainer" containerID="9d75bfa24b59187bf802d22b1dcc7b07b777e5f9ad2c8ea643145fce877e347d" Sep 10 12:10:01.317490 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:10:01.317423 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d75bfa24b59187bf802d22b1dcc7b07b777e5f9ad2c8ea643145fce877e347d\": container with ID starting with 9d75bfa24b59187bf802d22b1dcc7b07b777e5f9ad2c8ea643145fce877e347d not found: ID does not exist" containerID="9d75bfa24b59187bf802d22b1dcc7b07b777e5f9ad2c8ea643145fce877e347d" Sep 10 12:10:01.317490 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.317462 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d75bfa24b59187bf802d22b1dcc7b07b777e5f9ad2c8ea643145fce877e347d"} err="failed to get container status \"9d75bfa24b59187bf802d22b1dcc7b07b777e5f9ad2c8ea643145fce877e347d\": rpc error: code = NotFound desc = could not find container \"9d75bfa24b59187bf802d22b1dcc7b07b777e5f9ad2c8ea643145fce877e347d\": container with ID starting with 9d75bfa24b59187bf802d22b1dcc7b07b777e5f9ad2c8ea643145fce877e347d not found: ID does not exist" Sep 10 12:10:01.371215 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371190 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-istio-data\") pod \"851f1d92-74cc-456f-a059-913d2368e85d\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " Sep 10 12:10:01.371391 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371225 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-istio-envoy\") pod \"851f1d92-74cc-456f-a059-913d2368e85d\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " Sep 10 12:10:01.371391 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371261 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-workload-certs\") pod \"851f1d92-74cc-456f-a059-913d2368e85d\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " Sep 10 12:10:01.371391 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371286 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851f1d92-74cc-456f-a059-913d2368e85d-istiod-ca-cert\") pod \"851f1d92-74cc-456f-a059-913d2368e85d\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " Sep 10 12:10:01.371391 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371306 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjxtb\" (UniqueName: \"kubernetes.io/projected/851f1d92-74cc-456f-a059-913d2368e85d-kube-api-access-tjxtb\") pod \"851f1d92-74cc-456f-a059-913d2368e85d\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " Sep 10 12:10:01.371391 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371372 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-credential-socket\") pod \"851f1d92-74cc-456f-a059-913d2368e85d\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " Sep 10 12:10:01.371660 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371434 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851f1d92-74cc-456f-a059-913d2368e85d-istio-podinfo\") pod \"851f1d92-74cc-456f-a059-913d2368e85d\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " Sep 10 12:10:01.371660 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371459 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-workload-socket\") pod \"851f1d92-74cc-456f-a059-913d2368e85d\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " Sep 10 12:10:01.371660 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371479 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-istio-data" (OuterVolumeSpecName: "istio-data") pod "851f1d92-74cc-456f-a059-913d2368e85d" (UID: "851f1d92-74cc-456f-a059-913d2368e85d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:10:01.371660 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371501 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851f1d92-74cc-456f-a059-913d2368e85d-istio-token\") pod \"851f1d92-74cc-456f-a059-913d2368e85d\" (UID: \"851f1d92-74cc-456f-a059-913d2368e85d\") " Sep 10 12:10:01.371660 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371554 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "851f1d92-74cc-456f-a059-913d2368e85d" (UID: "851f1d92-74cc-456f-a059-913d2368e85d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:10:01.371838 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371737 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851f1d92-74cc-456f-a059-913d2368e85d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "851f1d92-74cc-456f-a059-913d2368e85d" (UID: "851f1d92-74cc-456f-a059-913d2368e85d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:10:01.371961 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371943 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.372022 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371968 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.372022 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.371980 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/851f1d92-74cc-456f-a059-913d2368e85d-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.372308 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.372271 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "851f1d92-74cc-456f-a059-913d2368e85d" (UID: "851f1d92-74cc-456f-a059-913d2368e85d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:10:01.372451 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.372427 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "851f1d92-74cc-456f-a059-913d2368e85d" (UID: "851f1d92-74cc-456f-a059-913d2368e85d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:10:01.374473 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.374452 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851f1d92-74cc-456f-a059-913d2368e85d-kube-api-access-tjxtb" (OuterVolumeSpecName: "kube-api-access-tjxtb") pod "851f1d92-74cc-456f-a059-913d2368e85d" (UID: "851f1d92-74cc-456f-a059-913d2368e85d"). InnerVolumeSpecName "kube-api-access-tjxtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:10:01.374584 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.374560 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "851f1d92-74cc-456f-a059-913d2368e85d" (UID: "851f1d92-74cc-456f-a059-913d2368e85d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:10:01.374652 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.374578 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/851f1d92-74cc-456f-a059-913d2368e85d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "851f1d92-74cc-456f-a059-913d2368e85d" (UID: "851f1d92-74cc-456f-a059-913d2368e85d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:10:01.374652 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.374603 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851f1d92-74cc-456f-a059-913d2368e85d-istio-token" (OuterVolumeSpecName: "istio-token") pod "851f1d92-74cc-456f-a059-913d2368e85d" (UID: "851f1d92-74cc-456f-a059-913d2368e85d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:10:01.402744 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.402718 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:10:01.472558 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.472476 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.472558 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.472503 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/851f1d92-74cc-456f-a059-913d2368e85d-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.472558 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.472512 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.472558 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.472522 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/851f1d92-74cc-456f-a059-913d2368e85d-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.472558 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.472535 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/851f1d92-74cc-456f-a059-913d2368e85d-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.472558 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.472546 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjxtb\" (UniqueName: \"kubernetes.io/projected/851f1d92-74cc-456f-a059-913d2368e85d-kube-api-access-tjxtb\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.573096 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573065 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-credential-socket\") pod \"db3c479c-8910-473c-a499-8b9e8d8d10b6\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " Sep 10 12:10:01.573096 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573112 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrttd\" (UniqueName: \"kubernetes.io/projected/db3c479c-8910-473c-a499-8b9e8d8d10b6-kube-api-access-hrttd\") pod \"db3c479c-8910-473c-a499-8b9e8d8d10b6\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " Sep 10 12:10:01.573377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573133 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-workload-certs\") pod \"db3c479c-8910-473c-a499-8b9e8d8d10b6\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " Sep 10 12:10:01.573377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573194 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-podinfo\") pod \"db3c479c-8910-473c-a499-8b9e8d8d10b6\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " Sep 10 12:10:01.573377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573214 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-token\") pod \"db3c479c-8910-473c-a499-8b9e8d8d10b6\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " Sep 10 12:10:01.573377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573227 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-envoy\") pod \"db3c479c-8910-473c-a499-8b9e8d8d10b6\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " Sep 10 12:10:01.573377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573254 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-workload-socket\") pod \"db3c479c-8910-473c-a499-8b9e8d8d10b6\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " Sep 10 12:10:01.573377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573329 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3c479c-8910-473c-a499-8b9e8d8d10b6-istiod-ca-cert\") pod \"db3c479c-8910-473c-a499-8b9e8d8d10b6\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " Sep 10 12:10:01.573377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573359 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-data\") pod \"db3c479c-8910-473c-a499-8b9e8d8d10b6\" (UID: \"db3c479c-8910-473c-a499-8b9e8d8d10b6\") " Sep 10 12:10:01.573727 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573474 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "db3c479c-8910-473c-a499-8b9e8d8d10b6" (UID: "db3c479c-8910-473c-a499-8b9e8d8d10b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:10:01.573727 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573592 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "db3c479c-8910-473c-a499-8b9e8d8d10b6" (UID: "db3c479c-8910-473c-a499-8b9e8d8d10b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:10:01.573823 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573748 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "db3c479c-8910-473c-a499-8b9e8d8d10b6" (UID: "db3c479c-8910-473c-a499-8b9e8d8d10b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:10:01.573883 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573867 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.573941 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573889 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.573941 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573904 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.574029 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.573985 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "db3c479c-8910-473c-a499-8b9e8d8d10b6" (UID: "db3c479c-8910-473c-a499-8b9e8d8d10b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:10:01.574070 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.574047 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3c479c-8910-473c-a499-8b9e8d8d10b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "db3c479c-8910-473c-a499-8b9e8d8d10b6" (UID: "db3c479c-8910-473c-a499-8b9e8d8d10b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:10:01.575826 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.575799 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "db3c479c-8910-473c-a499-8b9e8d8d10b6" (UID: "db3c479c-8910-473c-a499-8b9e8d8d10b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:10:01.575935 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.575876 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "db3c479c-8910-473c-a499-8b9e8d8d10b6" (UID: "db3c479c-8910-473c-a499-8b9e8d8d10b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:10:01.576029 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.576011 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "db3c479c-8910-473c-a499-8b9e8d8d10b6" (UID: "db3c479c-8910-473c-a499-8b9e8d8d10b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:10:01.576029 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.576021 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3c479c-8910-473c-a499-8b9e8d8d10b6-kube-api-access-hrttd" (OuterVolumeSpecName: "kube-api-access-hrttd") pod "db3c479c-8910-473c-a499-8b9e8d8d10b6" (UID: "db3c479c-8910-473c-a499-8b9e8d8d10b6"). InnerVolumeSpecName "kube-api-access-hrttd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:10:01.621956 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.621918 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5"] Sep 10 12:10:01.629840 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.629809 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-c8a24e-openshift-default-85fbd795f8-qs9m5"] Sep 10 12:10:01.675311 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.675277 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrttd\" (UniqueName: \"kubernetes.io/projected/db3c479c-8910-473c-a499-8b9e8d8d10b6-kube-api-access-hrttd\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.675311 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.675306 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.675311 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.675317 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.675526 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.675327 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.675526 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.675335 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db3c479c-8910-473c-a499-8b9e8d8d10b6-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:01.675526 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:01.675342 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db3c479c-8910-473c-a499-8b9e8d8d10b6-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:10:02.299521 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:02.299445 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" event={"ID":"db3c479c-8910-473c-a499-8b9e8d8d10b6","Type":"ContainerDied","Data":"59097d919e45b9bb674e2f78286c0a6fb532cdfe2f3e48a7a1ee14d1eb066320"} Sep 10 12:10:02.299521 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:02.299469 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc" Sep 10 12:10:02.299521 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:02.299503 2582 scope.go:117] "RemoveContainer" containerID="1f6219b7814fed0acb2daddd08c8d3defa7b79d8c7366c7739ad0706c6f9e67a" Sep 10 12:10:02.337239 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:02.337204 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc"] Sep 10 12:10:02.348677 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:02.348650 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-425dc7-openshift-default-58899c88f8-4dvvc"] Sep 10 12:10:02.630206 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:02.630112 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="851f1d92-74cc-456f-a059-913d2368e85d" path="/var/lib/kubelet/pods/851f1d92-74cc-456f-a059-913d2368e85d/volumes" Sep 10 12:10:02.630557 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:02.630544 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" path="/var/lib/kubelet/pods/db3c479c-8910-473c-a499-8b9e8d8d10b6/volumes" Sep 10 12:10:05.086068 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:05.086028 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:10:17.990077 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:17.990043 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:10:25.883976 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.883946 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2"] Sep 10 12:10:25.884374 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884360 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerName="istio-proxy" Sep 10 12:10:25.884419 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884376 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerName="istio-proxy" Sep 10 12:10:25.884419 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884398 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" Sep 10 12:10:25.884419 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884404 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" Sep 10 12:10:25.884419 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884410 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851f1d92-74cc-456f-a059-913d2368e85d" containerName="istio-proxy" Sep 10 12:10:25.884419 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884415 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="851f1d92-74cc-456f-a059-913d2368e85d" containerName="istio-proxy" Sep 10 12:10:25.884576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884421 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24fa3df3-a321-4677-96ac-97f1d6d60b6f" containerName="revoke-keys" Sep 10 12:10:25.884576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884426 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="24fa3df3-a321-4677-96ac-97f1d6d60b6f" containerName="revoke-keys" Sep 10 12:10:25.884576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884433 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" Sep 10 12:10:25.884576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884438 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f911bf6-0c4c-4787-95fd-758415991c41" containerName="istio-proxy" Sep 10 12:10:25.884576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884446 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" Sep 10 12:10:25.884576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884451 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" Sep 10 12:10:25.884576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884504 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerName="istio-proxy" Sep 10 12:10:25.884576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884512 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="851f1d92-74cc-456f-a059-913d2368e85d" containerName="istio-proxy" Sep 10 12:10:25.884576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884518 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="24fa3df3-a321-4677-96ac-97f1d6d60b6f" containerName="revoke-keys" Sep 10 12:10:25.884576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884527 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" Sep 10 12:10:25.884576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.884531 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f5472cb-8a14-47be-a9aa-a733c366169e" containerName="istio-proxy" Sep 10 12:10:25.889191 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.889171 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:25.894801 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.894775 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-7ed5ac-openshift-default-dockercfg-vt2sc\"" Sep 10 12:10:25.908860 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:25.908837 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2"] Sep 10 12:10:26.012066 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.012029 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-podinfo\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.012288 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.012170 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnf9b\" (UniqueName: \"kubernetes.io/projected/eba69eaf-fc89-4c26-889a-c45223e69cc7-kube-api-access-fnf9b\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.012288 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.012212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-token\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.012288 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.012254 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-workload-socket\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.012288 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.012275 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba69eaf-fc89-4c26-889a-c45223e69cc7-istiod-ca-cert\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.012505 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.012300 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-credential-socket\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.012505 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.012326 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-envoy\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.012505 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.012360 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-workload-certs\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.012505 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.012397 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-data\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.113828 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.113794 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-data\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.114018 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.113838 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-podinfo\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.114018 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.113984 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnf9b\" (UniqueName: \"kubernetes.io/projected/eba69eaf-fc89-4c26-889a-c45223e69cc7-kube-api-access-fnf9b\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.114181 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.114028 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-token\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.114181 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.114080 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-workload-socket\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.114181 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.114105 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba69eaf-fc89-4c26-889a-c45223e69cc7-istiod-ca-cert\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.114181 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.114137 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-credential-socket\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.114412 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.114178 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-data\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.114412 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.114195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-envoy\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.114412 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.114249 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-workload-certs\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.114522 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.114410 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-workload-socket\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.114522 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.114493 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-credential-socket\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.114738 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.114718 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-workload-certs\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.114963 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.114947 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba69eaf-fc89-4c26-889a-c45223e69cc7-istiod-ca-cert\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.116440 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.116415 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-envoy\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.116583 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.116469 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-podinfo\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.129066 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.129042 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnf9b\" (UniqueName: \"kubernetes.io/projected/eba69eaf-fc89-4c26-889a-c45223e69cc7-kube-api-access-fnf9b\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.131277 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.131260 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-token\") pod \"e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.201405 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.201371 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:26.488253 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.488231 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2"] Sep 10 12:10:26.490212 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:10:26.490182 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba69eaf_fc89_4c26_889a_c45223e69cc7.slice/crio-af171c9c3f9b4a65394fc3940e760ab0e80d425d1432690c7ef0beb9166fac3a WatchSource:0}: Error finding container af171c9c3f9b4a65394fc3940e760ab0e80d425d1432690c7ef0beb9166fac3a: Status 404 returned error can't find the container with id af171c9c3f9b4a65394fc3940e760ab0e80d425d1432690c7ef0beb9166fac3a Sep 10 12:10:26.492294 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.492260 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:10:26.492384 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.492329 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:10:26.492384 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:26.492360 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:10:27.444830 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:27.444793 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" event={"ID":"eba69eaf-fc89-4c26-889a-c45223e69cc7","Type":"ContainerStarted","Data":"38b9e8f2b1e0589dc1917e31676ad2b7abc8ca76777fd8dde17d104ca8cb3af7"} Sep 10 12:10:27.445222 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:27.444844 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" event={"ID":"eba69eaf-fc89-4c26-889a-c45223e69cc7","Type":"ContainerStarted","Data":"af171c9c3f9b4a65394fc3940e760ab0e80d425d1432690c7ef0beb9166fac3a"} Sep 10 12:10:27.477917 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:27.477867 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" podStartSLOduration=2.477850631 podStartE2EDuration="2.477850631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:27.474907259 +0000 UTC m=+1135.354296690" watchObservedRunningTime="2026-09-10 12:10:27.477850631 +0000 UTC m=+1135.357240062" Sep 10 12:10:28.201841 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:28.201808 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:28.206846 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:28.206821 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:28.451001 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:28.450968 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:28.451910 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:28.451854 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:35.562489 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:35.562447 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:10:38.016173 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.011872 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt"] Sep 10 12:10:38.016173 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.012531 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerName="istio-proxy" Sep 10 12:10:38.016173 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.012548 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerName="istio-proxy" Sep 10 12:10:38.016173 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.012662 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3c479c-8910-473c-a499-8b9e8d8d10b6" containerName="istio-proxy" Sep 10 12:10:38.025448 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.025423 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.029003 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.028916 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c66562-openshift-default-dockercfg-5qsgt\"" Sep 10 12:10:38.033977 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.033949 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt"] Sep 10 12:10:38.131764 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.131722 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-istio-data\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.131764 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.131759 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac3a5983-e046-4956-aefa-23d9413f861a-istio-token\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.131996 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.131791 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-credential-socket\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.131996 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.131921 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-workload-socket\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.131996 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.131981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-workload-certs\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.132132 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.132047 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwsv8\" (UniqueName: \"kubernetes.io/projected/ac3a5983-e046-4956-aefa-23d9413f861a-kube-api-access-rwsv8\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.132132 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.132084 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-istio-envoy\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.132132 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.132111 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3a5983-e046-4956-aefa-23d9413f861a-istiod-ca-cert\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.132308 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.132196 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3a5983-e046-4956-aefa-23d9413f861a-istio-podinfo\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.233268 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.233226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-credential-socket\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.233475 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.233287 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-workload-socket\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.233475 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.233312 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-workload-certs\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.233475 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.233352 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwsv8\" (UniqueName: \"kubernetes.io/projected/ac3a5983-e046-4956-aefa-23d9413f861a-kube-api-access-rwsv8\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.233475 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.233374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-istio-envoy\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.233475 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.233391 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3a5983-e046-4956-aefa-23d9413f861a-istiod-ca-cert\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.233475 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.233414 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3a5983-e046-4956-aefa-23d9413f861a-istio-podinfo\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.233475 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.233470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-istio-data\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.233875 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.233497 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac3a5983-e046-4956-aefa-23d9413f861a-istio-token\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.233875 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.233722 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-credential-socket\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.233875 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.233748 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-workload-certs\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.233875 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.233764 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-workload-socket\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.234074 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.233984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-istio-data\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.234259 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.234240 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3a5983-e046-4956-aefa-23d9413f861a-istiod-ca-cert\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.235869 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.235836 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-istio-envoy\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.236104 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.236084 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3a5983-e046-4956-aefa-23d9413f861a-istio-podinfo\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.243100 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.243076 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac3a5983-e046-4956-aefa-23d9413f861a-istio-token\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.244673 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.244655 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwsv8\" (UniqueName: \"kubernetes.io/projected/ac3a5983-e046-4956-aefa-23d9413f861a-kube-api-access-rwsv8\") pod \"e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.339354 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.339265 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:38.646912 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.646888 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt"] Sep 10 12:10:38.647731 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:10:38.647704 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3a5983_e046_4956_aefa_23d9413f861a.slice/crio-99b39310ceb58b8b2d1eac62e2c014ed04090ff50b7ffaf15b86010ee41b2699 WatchSource:0}: Error finding container 99b39310ceb58b8b2d1eac62e2c014ed04090ff50b7ffaf15b86010ee41b2699: Status 404 returned error can't find the container with id 99b39310ceb58b8b2d1eac62e2c014ed04090ff50b7ffaf15b86010ee41b2699 Sep 10 12:10:38.649914 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.649879 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:10:38.650031 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.649959 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:10:38.650031 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:38.650001 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:10:39.531750 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:39.531716 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" event={"ID":"ac3a5983-e046-4956-aefa-23d9413f861a","Type":"ContainerStarted","Data":"e67568c6f071d99f6699818852be84cdad3871a81fac8fc6231831578c82aecf"} Sep 10 12:10:39.532137 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:39.531764 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" event={"ID":"ac3a5983-e046-4956-aefa-23d9413f861a","Type":"ContainerStarted","Data":"99b39310ceb58b8b2d1eac62e2c014ed04090ff50b7ffaf15b86010ee41b2699"} Sep 10 12:10:39.560880 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:39.560835 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" podStartSLOduration=2.560822155 podStartE2EDuration="2.560822155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:39.55651376 +0000 UTC m=+1147.435903192" watchObservedRunningTime="2026-09-10 12:10:39.560822155 +0000 UTC m=+1147.440211587" Sep 10 12:10:40.340288 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:40.340247 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:40.345739 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:40.345714 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:40.537896 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:40.537865 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:40.538806 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:40.538790 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:10:44.492270 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:44.492232 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 12:10:44.492670 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:44.492289 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 12:10:46.576603 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:46.576571 2582 generic.go:356] "Generic (PLEG): container finished" podID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerID="38b9e8f2b1e0589dc1917e31676ad2b7abc8ca76777fd8dde17d104ca8cb3af7" exitCode=0 Sep 10 12:10:46.577016 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:46.576647 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" event={"ID":"eba69eaf-fc89-4c26-889a-c45223e69cc7","Type":"ContainerDied","Data":"38b9e8f2b1e0589dc1917e31676ad2b7abc8ca76777fd8dde17d104ca8cb3af7"} Sep 10 12:10:46.577071 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:46.577056 2582 scope.go:117] "RemoveContainer" containerID="38b9e8f2b1e0589dc1917e31676ad2b7abc8ca76777fd8dde17d104ca8cb3af7" Sep 10 12:10:46.578124 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:46.578098 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:10:46.578201 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:46.578186 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:10:46.578254 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:46.578242 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:10:47.585247 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:47.585210 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" event={"ID":"eba69eaf-fc89-4c26-889a-c45223e69cc7","Type":"ContainerStarted","Data":"b149175f6b64d0d1abd1767a2f6942ac826672ef88fee96c6af3255191e9a8ce"} Sep 10 12:10:48.202486 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:48.202454 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:48.202854 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:48.202769 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 10 12:10:48.203023 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:48.203000 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 10 12:10:49.202160 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:49.202102 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 10 12:10:49.202657 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:49.202222 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 10 12:10:50.202642 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:50.202604 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 10 12:10:50.203137 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:50.202692 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 10 12:10:51.202783 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:51.202737 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 10 12:10:51.203377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:51.202806 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 10 12:10:51.298320 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:51.298279 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:10:52.202289 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:52.202249 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 10 12:10:52.202576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:52.202545 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 10 12:10:53.217786 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:53.217734 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:53.218928 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:53.218897 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:10:53.223027 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:10:53.223003 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:11:02.681955 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:02.681925 2582 generic.go:356] "Generic (PLEG): container finished" podID="ac3a5983-e046-4956-aefa-23d9413f861a" containerID="e67568c6f071d99f6699818852be84cdad3871a81fac8fc6231831578c82aecf" exitCode=0 Sep 10 12:11:02.682509 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:02.682002 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" event={"ID":"ac3a5983-e046-4956-aefa-23d9413f861a","Type":"ContainerDied","Data":"e67568c6f071d99f6699818852be84cdad3871a81fac8fc6231831578c82aecf"} Sep 10 12:11:02.682578 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:02.682555 2582 scope.go:117] "RemoveContainer" containerID="e67568c6f071d99f6699818852be84cdad3871a81fac8fc6231831578c82aecf" Sep 10 12:11:02.683685 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:02.683658 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:11:02.683781 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:02.683739 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:11:02.683850 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:02.683786 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:11:03.689713 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:03.689674 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" event={"ID":"ac3a5983-e046-4956-aefa-23d9413f861a","Type":"ContainerStarted","Data":"835059695a73a84dea91fa560eb45199eb428e94b0af1e9d63d395ec5def9f0a"} Sep 10 12:11:04.340135 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:04.340102 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:11:04.340418 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:04.340373 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 10 12:11:04.340467 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:04.340430 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" podUID="ac3a5983-e046-4956-aefa-23d9413f861a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 10 12:11:05.340626 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:05.340589 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 10 12:11:05.341028 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:05.340658 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" podUID="ac3a5983-e046-4956-aefa-23d9413f861a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 10 12:11:06.340438 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:06.340404 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 10 12:11:06.340622 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:06.340456 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" podUID="ac3a5983-e046-4956-aefa-23d9413f861a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 10 12:11:07.258480 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:07.258443 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7958-139715767523344"] Sep 10 12:11:07.262025 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:07.262007 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:07.265435 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:07.265410 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7cc6f\"" Sep 10 12:11:07.267271 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:07.267250 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7958-139715767523344"] Sep 10 12:11:07.340670 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:07.340628 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 10 12:11:07.340845 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:07.340689 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" podUID="ac3a5983-e046-4956-aefa-23d9413f861a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 10 12:11:07.442059 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:07.442023 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kkg7\" (UniqueName: \"kubernetes.io/projected/9a0501f4-a293-451e-a243-99504d7dcfc6-kube-api-access-9kkg7\") pod \"test-curl-7958-139715767523344\" (UID: \"9a0501f4-a293-451e-a243-99504d7dcfc6\") " pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:07.543553 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:07.543456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kkg7\" (UniqueName: \"kubernetes.io/projected/9a0501f4-a293-451e-a243-99504d7dcfc6-kube-api-access-9kkg7\") pod \"test-curl-7958-139715767523344\" (UID: \"9a0501f4-a293-451e-a243-99504d7dcfc6\") " pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:07.552756 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:07.552730 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kkg7\" (UniqueName: \"kubernetes.io/projected/9a0501f4-a293-451e-a243-99504d7dcfc6-kube-api-access-9kkg7\") pod \"test-curl-7958-139715767523344\" (UID: \"9a0501f4-a293-451e-a243-99504d7dcfc6\") " pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:07.572610 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:07.572583 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:07.861989 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:07.861967 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7958-139715767523344"] Sep 10 12:11:07.864186 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:11:07.864125 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a0501f4_a293_451e_a243_99504d7dcfc6.slice/crio-55cd88995748e16bf62740832224fa672a5722526a18d9d1199036209e76f6a4 WatchSource:0}: Error finding container 55cd88995748e16bf62740832224fa672a5722526a18d9d1199036209e76f6a4: Status 404 returned error can't find the container with id 55cd88995748e16bf62740832224fa672a5722526a18d9d1199036209e76f6a4 Sep 10 12:11:07.865475 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:07.865455 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 12:11:08.339807 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:08.339767 2582 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 10 12:11:08.340235 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:08.339834 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" podUID="ac3a5983-e046-4956-aefa-23d9413f861a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 10 12:11:08.731459 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:08.731420 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7958-139715767523344" event={"ID":"9a0501f4-a293-451e-a243-99504d7dcfc6","Type":"ContainerStarted","Data":"55cd88995748e16bf62740832224fa672a5722526a18d9d1199036209e76f6a4"} Sep 10 12:11:09.343880 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:09.343853 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:11:09.344317 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:09.344102 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:11:09.345023 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:09.345004 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:11:09.740925 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:09.740893 2582 generic.go:356] "Generic (PLEG): container finished" podID="9a0501f4-a293-451e-a243-99504d7dcfc6" containerID="e39d1388a7f27c73dcbe440a9d32d4a4882bff099ec2449ef44973aec09a5370" exitCode=0 Sep 10 12:11:09.741101 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:09.740981 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7958-139715767523344" event={"ID":"9a0501f4-a293-451e-a243-99504d7dcfc6","Type":"ContainerDied","Data":"e39d1388a7f27c73dcbe440a9d32d4a4882bff099ec2449ef44973aec09a5370"} Sep 10 12:11:10.882664 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:10.882641 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:10.924826 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:10.924802 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7958-139715767523344_9a0501f4-a293-451e-a243-99504d7dcfc6/test-curl-7958-139715767523344/0.log" Sep 10 12:11:10.949092 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:10.949069 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7958-139715767523344"] Sep 10 12:11:10.954377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:10.954357 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7958-139715767523344"] Sep 10 12:11:10.974543 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:10.974520 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kkg7\" (UniqueName: \"kubernetes.io/projected/9a0501f4-a293-451e-a243-99504d7dcfc6-kube-api-access-9kkg7\") pod \"9a0501f4-a293-451e-a243-99504d7dcfc6\" (UID: \"9a0501f4-a293-451e-a243-99504d7dcfc6\") " Sep 10 12:11:10.976569 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:10.976533 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0501f4-a293-451e-a243-99504d7dcfc6-kube-api-access-9kkg7" (OuterVolumeSpecName: "kube-api-access-9kkg7") pod "9a0501f4-a293-451e-a243-99504d7dcfc6" (UID: "9a0501f4-a293-451e-a243-99504d7dcfc6"). InnerVolumeSpecName "kube-api-access-9kkg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:11:11.075912 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.075818 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kkg7\" (UniqueName: \"kubernetes.io/projected/9a0501f4-a293-451e-a243-99504d7dcfc6-kube-api-access-9kkg7\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:11:11.078270 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.078243 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7958-139715767523344"] Sep 10 12:11:11.078657 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.078644 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a0501f4-a293-451e-a243-99504d7dcfc6" containerName="test-curl-7958-139715767523344" Sep 10 12:11:11.078700 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.078658 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a0501f4-a293-451e-a243-99504d7dcfc6" containerName="test-curl-7958-139715767523344" Sep 10 12:11:11.078742 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.078726 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a0501f4-a293-451e-a243-99504d7dcfc6" containerName="test-curl-7958-139715767523344" Sep 10 12:11:11.081646 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.081632 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:11.090062 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.090033 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7958-139715767523344"] Sep 10 12:11:11.176308 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.176277 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rht52\" (UniqueName: \"kubernetes.io/projected/b5366b90-9d7e-45c1-a723-ed27237bab8d-kube-api-access-rht52\") pod \"test-curl-7958-139715767523344\" (UID: \"b5366b90-9d7e-45c1-a723-ed27237bab8d\") " pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:11.277268 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.277239 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rht52\" (UniqueName: \"kubernetes.io/projected/b5366b90-9d7e-45c1-a723-ed27237bab8d-kube-api-access-rht52\") pod \"test-curl-7958-139715767523344\" (UID: \"b5366b90-9d7e-45c1-a723-ed27237bab8d\") " pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:11.286577 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.286550 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rht52\" (UniqueName: \"kubernetes.io/projected/b5366b90-9d7e-45c1-a723-ed27237bab8d-kube-api-access-rht52\") pod \"test-curl-7958-139715767523344\" (UID: \"b5366b90-9d7e-45c1-a723-ed27237bab8d\") " pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:11.391127 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.391045 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:11.661513 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.661490 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7958-139715767523344"] Sep 10 12:11:11.663841 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:11:11.663815 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5366b90_9d7e_45c1_a723_ed27237bab8d.slice/crio-a259f65402ccf1513904a24171bf1a62cb4a7e6a0fba69c2f07f1326c75b284f WatchSource:0}: Error finding container a259f65402ccf1513904a24171bf1a62cb4a7e6a0fba69c2f07f1326c75b284f: Status 404 returned error can't find the container with id a259f65402ccf1513904a24171bf1a62cb4a7e6a0fba69c2f07f1326c75b284f Sep 10 12:11:11.753022 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.752995 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7958-139715767523344" event={"ID":"b5366b90-9d7e-45c1-a723-ed27237bab8d","Type":"ContainerStarted","Data":"a259f65402ccf1513904a24171bf1a62cb4a7e6a0fba69c2f07f1326c75b284f"} Sep 10 12:11:11.754289 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.754264 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55cd88995748e16bf62740832224fa672a5722526a18d9d1199036209e76f6a4" Sep 10 12:11:11.754389 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.754294 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:11.765724 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:11.765696 2582 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7958-139715767523344" oldPodUID="9a0501f4-a293-451e-a243-99504d7dcfc6" podUID="b5366b90-9d7e-45c1-a723-ed27237bab8d" Sep 10 12:11:12.629861 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:12.629784 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a0501f4-a293-451e-a243-99504d7dcfc6" path="/var/lib/kubelet/pods/9a0501f4-a293-451e-a243-99504d7dcfc6/volumes" Sep 10 12:11:12.761660 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:12.761631 2582 generic.go:356] "Generic (PLEG): container finished" podID="b5366b90-9d7e-45c1-a723-ed27237bab8d" containerID="455400dd8e1a00c0da721a1fcc4d4f78a983f90a7c3f4f535f704cbd5cdaa9fd" exitCode=0 Sep 10 12:11:12.761842 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:12.761724 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7958-139715767523344" event={"ID":"b5366b90-9d7e-45c1-a723-ed27237bab8d","Type":"ContainerDied","Data":"455400dd8e1a00c0da721a1fcc4d4f78a983f90a7c3f4f535f704cbd5cdaa9fd"} Sep 10 12:11:13.904458 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:13.904434 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:13.931964 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:13.931928 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7958-139715767523344_b5366b90-9d7e-45c1-a723-ed27237bab8d/test-curl-7958-139715767523344/0.log" Sep 10 12:11:13.956542 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:13.956515 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7958-139715767523344"] Sep 10 12:11:13.960830 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:13.960811 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7958-139715767523344"] Sep 10 12:11:14.005741 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.005710 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rht52\" (UniqueName: \"kubernetes.io/projected/b5366b90-9d7e-45c1-a723-ed27237bab8d-kube-api-access-rht52\") pod \"b5366b90-9d7e-45c1-a723-ed27237bab8d\" (UID: \"b5366b90-9d7e-45c1-a723-ed27237bab8d\") " Sep 10 12:11:14.007856 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.007829 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5366b90-9d7e-45c1-a723-ed27237bab8d-kube-api-access-rht52" (OuterVolumeSpecName: "kube-api-access-rht52") pod "b5366b90-9d7e-45c1-a723-ed27237bab8d" (UID: "b5366b90-9d7e-45c1-a723-ed27237bab8d"). InnerVolumeSpecName "kube-api-access-rht52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:11:14.095504 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.095416 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7958-139715767523920"] Sep 10 12:11:14.095842 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.095829 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5366b90-9d7e-45c1-a723-ed27237bab8d" containerName="test-curl-7958-139715767523344" Sep 10 12:11:14.095887 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.095843 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5366b90-9d7e-45c1-a723-ed27237bab8d" containerName="test-curl-7958-139715767523344" Sep 10 12:11:14.095925 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.095921 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5366b90-9d7e-45c1-a723-ed27237bab8d" containerName="test-curl-7958-139715767523344" Sep 10 12:11:14.098968 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.098952 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:14.103946 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.103921 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7958-139715767523920"] Sep 10 12:11:14.106564 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.106538 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rht52\" (UniqueName: \"kubernetes.io/projected/b5366b90-9d7e-45c1-a723-ed27237bab8d-kube-api-access-rht52\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:11:14.207541 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.207504 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gm6\" (UniqueName: \"kubernetes.io/projected/be644332-41fd-4cba-91cf-d7e92ae34c53-kube-api-access-p5gm6\") pod \"test-curl-7958-139715767523920\" (UID: \"be644332-41fd-4cba-91cf-d7e92ae34c53\") " pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:14.308297 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.308266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gm6\" (UniqueName: \"kubernetes.io/projected/be644332-41fd-4cba-91cf-d7e92ae34c53-kube-api-access-p5gm6\") pod \"test-curl-7958-139715767523920\" (UID: \"be644332-41fd-4cba-91cf-d7e92ae34c53\") " pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:14.317575 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.317551 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gm6\" (UniqueName: \"kubernetes.io/projected/be644332-41fd-4cba-91cf-d7e92ae34c53-kube-api-access-p5gm6\") pod \"test-curl-7958-139715767523920\" (UID: \"be644332-41fd-4cba-91cf-d7e92ae34c53\") " pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:14.409316 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.409235 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:14.630315 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.630281 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5366b90-9d7e-45c1-a723-ed27237bab8d" path="/var/lib/kubelet/pods/b5366b90-9d7e-45c1-a723-ed27237bab8d/volumes" Sep 10 12:11:14.679905 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.679885 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7958-139715767523920"] Sep 10 12:11:14.681852 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:11:14.681828 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe644332_41fd_4cba_91cf_d7e92ae34c53.slice/crio-a3ce18869ca303982a04c08271d55805f8237c6e884762e310de41b82f875709 WatchSource:0}: Error finding container a3ce18869ca303982a04c08271d55805f8237c6e884762e310de41b82f875709: Status 404 returned error can't find the container with id a3ce18869ca303982a04c08271d55805f8237c6e884762e310de41b82f875709 Sep 10 12:11:14.774313 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.774278 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7958-139715767523920" event={"ID":"be644332-41fd-4cba-91cf-d7e92ae34c53","Type":"ContainerStarted","Data":"a3ce18869ca303982a04c08271d55805f8237c6e884762e310de41b82f875709"} Sep 10 12:11:14.775417 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.775401 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523344" Sep 10 12:11:14.775511 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:14.775431 2582 scope.go:117] "RemoveContainer" containerID="455400dd8e1a00c0da721a1fcc4d4f78a983f90a7c3f4f535f704cbd5cdaa9fd" Sep 10 12:11:16.789580 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:16.789549 2582 generic.go:356] "Generic (PLEG): container finished" podID="be644332-41fd-4cba-91cf-d7e92ae34c53" containerID="8629a0f95e32e6b6440469827fcf546d67f2f29ec10b111e836e08fbaea7a154" exitCode=0 Sep 10 12:11:16.789974 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:16.789593 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7958-139715767523920" event={"ID":"be644332-41fd-4cba-91cf-d7e92ae34c53","Type":"ContainerDied","Data":"8629a0f95e32e6b6440469827fcf546d67f2f29ec10b111e836e08fbaea7a154"} Sep 10 12:11:17.933847 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:17.933823 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:17.955822 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:17.955800 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7958-139715767523920_be644332-41fd-4cba-91cf-d7e92ae34c53/test-curl-7958-139715767523920/0.log" Sep 10 12:11:17.981723 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:17.981700 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7958-139715767523920"] Sep 10 12:11:17.986156 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:17.986122 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7958-139715767523920"] Sep 10 12:11:18.044338 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.044317 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5gm6\" (UniqueName: \"kubernetes.io/projected/be644332-41fd-4cba-91cf-d7e92ae34c53-kube-api-access-p5gm6\") pod \"be644332-41fd-4cba-91cf-d7e92ae34c53\" (UID: \"be644332-41fd-4cba-91cf-d7e92ae34c53\") " Sep 10 12:11:18.046498 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.046437 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be644332-41fd-4cba-91cf-d7e92ae34c53-kube-api-access-p5gm6" (OuterVolumeSpecName: "kube-api-access-p5gm6") pod "be644332-41fd-4cba-91cf-d7e92ae34c53" (UID: "be644332-41fd-4cba-91cf-d7e92ae34c53"). InnerVolumeSpecName "kube-api-access-p5gm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:11:18.102172 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.102133 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7958-139715767523920"] Sep 10 12:11:18.102564 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.102550 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be644332-41fd-4cba-91cf-d7e92ae34c53" containerName="test-curl-7958-139715767523920" Sep 10 12:11:18.102612 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.102565 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="be644332-41fd-4cba-91cf-d7e92ae34c53" containerName="test-curl-7958-139715767523920" Sep 10 12:11:18.102648 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.102641 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="be644332-41fd-4cba-91cf-d7e92ae34c53" containerName="test-curl-7958-139715767523920" Sep 10 12:11:18.105662 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.105647 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:18.113456 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.113431 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7958-139715767523920"] Sep 10 12:11:18.145002 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.144970 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzx7l\" (UniqueName: \"kubernetes.io/projected/e8066db4-907f-4e93-89cb-ed100af89172-kube-api-access-lzx7l\") pod \"test-curl-7958-139715767523920\" (UID: \"e8066db4-907f-4e93-89cb-ed100af89172\") " pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:18.145170 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.145125 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5gm6\" (UniqueName: \"kubernetes.io/projected/be644332-41fd-4cba-91cf-d7e92ae34c53-kube-api-access-p5gm6\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:11:18.245979 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.245947 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzx7l\" (UniqueName: \"kubernetes.io/projected/e8066db4-907f-4e93-89cb-ed100af89172-kube-api-access-lzx7l\") pod \"test-curl-7958-139715767523920\" (UID: \"e8066db4-907f-4e93-89cb-ed100af89172\") " pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:18.255524 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.255498 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzx7l\" (UniqueName: \"kubernetes.io/projected/e8066db4-907f-4e93-89cb-ed100af89172-kube-api-access-lzx7l\") pod \"test-curl-7958-139715767523920\" (UID: \"e8066db4-907f-4e93-89cb-ed100af89172\") " pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:18.415868 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.415786 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:18.631702 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.631673 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be644332-41fd-4cba-91cf-d7e92ae34c53" path="/var/lib/kubelet/pods/be644332-41fd-4cba-91cf-d7e92ae34c53/volumes" Sep 10 12:11:18.696257 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.696231 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7958-139715767523920"] Sep 10 12:11:18.697216 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:11:18.697196 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8066db4_907f_4e93_89cb_ed100af89172.slice/crio-7b95ceac46f89d77c05bb15c59d715b953aecb0f3f2b32f37595f30322192778 WatchSource:0}: Error finding container 7b95ceac46f89d77c05bb15c59d715b953aecb0f3f2b32f37595f30322192778: Status 404 returned error can't find the container with id 7b95ceac46f89d77c05bb15c59d715b953aecb0f3f2b32f37595f30322192778 Sep 10 12:11:18.802426 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.802389 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7958-139715767523920" event={"ID":"e8066db4-907f-4e93-89cb-ed100af89172","Type":"ContainerStarted","Data":"7b95ceac46f89d77c05bb15c59d715b953aecb0f3f2b32f37595f30322192778"} Sep 10 12:11:18.803806 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.803774 2582 scope.go:117] "RemoveContainer" containerID="8629a0f95e32e6b6440469827fcf546d67f2f29ec10b111e836e08fbaea7a154" Sep 10 12:11:18.803928 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:18.803783 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:19.811929 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:19.811893 2582 generic.go:356] "Generic (PLEG): container finished" podID="e8066db4-907f-4e93-89cb-ed100af89172" containerID="ffd0cde60c46f8ce1cd4a986ce093012451958dd00104426666d86f800bc31f1" exitCode=0 Sep 10 12:11:19.812338 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:19.811961 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7958-139715767523920" event={"ID":"e8066db4-907f-4e93-89cb-ed100af89172","Type":"ContainerDied","Data":"ffd0cde60c46f8ce1cd4a986ce093012451958dd00104426666d86f800bc31f1"} Sep 10 12:11:20.952767 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:20.952742 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:20.975186 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:20.975114 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7958-139715767523920_e8066db4-907f-4e93-89cb-ed100af89172/test-curl-7958-139715767523920/0.log" Sep 10 12:11:21.001516 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:21.001488 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7958-139715767523920"] Sep 10 12:11:21.006893 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:21.006867 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7958-139715767523920"] Sep 10 12:11:21.073461 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:21.073378 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzx7l\" (UniqueName: \"kubernetes.io/projected/e8066db4-907f-4e93-89cb-ed100af89172-kube-api-access-lzx7l\") pod \"e8066db4-907f-4e93-89cb-ed100af89172\" (UID: \"e8066db4-907f-4e93-89cb-ed100af89172\") " Sep 10 12:11:21.075591 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:21.075565 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8066db4-907f-4e93-89cb-ed100af89172-kube-api-access-lzx7l" (OuterVolumeSpecName: "kube-api-access-lzx7l") pod "e8066db4-907f-4e93-89cb-ed100af89172" (UID: "e8066db4-907f-4e93-89cb-ed100af89172"). InnerVolumeSpecName "kube-api-access-lzx7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:11:21.175016 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:21.174985 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzx7l\" (UniqueName: \"kubernetes.io/projected/e8066db4-907f-4e93-89cb-ed100af89172-kube-api-access-lzx7l\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:11:21.823454 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:21.823428 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:21.823601 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:21.823458 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b95ceac46f89d77c05bb15c59d715b953aecb0f3f2b32f37595f30322192778" Sep 10 12:11:21.833219 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:11:21.833193 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7958-139715767523920\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="e8066db4-907f-4e93-89cb-ed100af89172" pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:22.000877 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.000846 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk"] Sep 10 12:11:22.001350 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.001337 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8066db4-907f-4e93-89cb-ed100af89172" containerName="test-curl-7958-139715767523920" Sep 10 12:11:22.001404 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.001352 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8066db4-907f-4e93-89cb-ed100af89172" containerName="test-curl-7958-139715767523920" Sep 10 12:11:22.001446 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.001431 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8066db4-907f-4e93-89cb-ed100af89172" containerName="test-curl-7958-139715767523920" Sep 10 12:11:22.004447 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.004432 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.007235 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.007214 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e74b3f66-openshift-default-dockercfg-grrzj\"" Sep 10 12:11:22.018521 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.018495 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk"] Sep 10 12:11:22.034243 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:11:22.034217 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7958-139715767523920\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="e8066db4-907f-4e93-89cb-ed100af89172" pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:22.082883 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.082791 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd72h\" (UniqueName: \"kubernetes.io/projected/f16f2c7e-7fc6-4159-b421-f1398514070b-kube-api-access-dd72h\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.082883 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.082831 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-workload-socket\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.082883 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.082859 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-credential-socket\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.083167 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.082929 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-envoy\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.083167 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.082970 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f2c7e-7fc6-4159-b421-f1398514070b-istiod-ca-cert\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.083167 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.083012 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-data\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.083167 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.083067 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-workload-certs\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.083167 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.083160 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-token\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.083349 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.083191 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-podinfo\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.183660 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.183626 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-workload-certs\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.183835 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.183685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-token\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.183835 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.183809 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-podinfo\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.183925 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.183854 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd72h\" (UniqueName: \"kubernetes.io/projected/f16f2c7e-7fc6-4159-b421-f1398514070b-kube-api-access-dd72h\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.183925 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.183886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-workload-socket\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.183925 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.183914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-credential-socket\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.184046 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.183968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-envoy\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.184046 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.184014 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f2c7e-7fc6-4159-b421-f1398514070b-istiod-ca-cert\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.184180 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.184049 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-workload-certs\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.184180 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.184061 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-data\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.184296 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.184257 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-workload-socket\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.184358 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.184346 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-data\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.184542 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.184521 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-credential-socket\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.184763 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.184740 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f2c7e-7fc6-4159-b421-f1398514070b-istiod-ca-cert\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.186425 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.186400 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-envoy\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.186536 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.186493 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-podinfo\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.192505 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.192483 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-token\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.192617 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.192601 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd72h\" (UniqueName: \"kubernetes.io/projected/f16f2c7e-7fc6-4159-b421-f1398514070b-kube-api-access-dd72h\") pod \"e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.315354 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.315317 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:22.592469 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.592442 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk"] Sep 10 12:11:22.594349 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:11:22.594301 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16f2c7e_7fc6_4159_b421_f1398514070b.slice/crio-f703901740cb696e0b6d705197134a40c4a2ba4a83c5a46b45f53456d81f8e1c WatchSource:0}: Error finding container f703901740cb696e0b6d705197134a40c4a2ba4a83c5a46b45f53456d81f8e1c: Status 404 returned error can't find the container with id f703901740cb696e0b6d705197134a40c4a2ba4a83c5a46b45f53456d81f8e1c Sep 10 12:11:22.597159 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.597104 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:11:22.597252 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.597226 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:11:22.597318 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.597298 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:11:22.628058 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:11:22.627997 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7958-139715767523920\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="e8066db4-907f-4e93-89cb-ed100af89172" pod="opendatahub/test-curl-7958-139715767523920" Sep 10 12:11:22.629943 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.629922 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8066db4-907f-4e93-89cb-ed100af89172" path="/var/lib/kubelet/pods/e8066db4-907f-4e93-89cb-ed100af89172/volumes" Sep 10 12:11:22.832561 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.832518 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" event={"ID":"f16f2c7e-7fc6-4159-b421-f1398514070b","Type":"ContainerStarted","Data":"2479cd718c09c05299b3e1c074ce1bed56d62b1452e0f3431abdd454feb69b80"} Sep 10 12:11:22.832726 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.832605 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" event={"ID":"f16f2c7e-7fc6-4159-b421-f1398514070b","Type":"ContainerStarted","Data":"f703901740cb696e0b6d705197134a40c4a2ba4a83c5a46b45f53456d81f8e1c"} Sep 10 12:11:22.854406 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:22.854357 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" podStartSLOduration=1.8543416700000002 podStartE2EDuration="1.85434167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:11:22.853661285 +0000 UTC m=+1190.733050741" watchObservedRunningTime="2026-09-10 12:11:22.85434167 +0000 UTC m=+1190.733731102" Sep 10 12:11:23.316363 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:23.316330 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:23.321559 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:23.321532 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:23.844160 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:23.844111 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:23.845087 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:23.845063 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:32.661447 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:32.661421 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 12:11:32.671387 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:32.671368 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 12:11:32.728788 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:32.728764 2582 scope.go:117] "RemoveContainer" containerID="6c0fa26f7ef98506ae47971e9e00a4cf765c35ebb6d7a346b0e34bd1d361c36a" Sep 10 12:11:32.737171 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:32.737136 2582 scope.go:117] "RemoveContainer" containerID="b1e0ad307bd4f9d6c4fca92a852f6505dedd3b737ce0d827e2eebab373316054" Sep 10 12:11:33.578280 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:33.578236 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:11:43.968179 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:43.968126 2582 generic.go:356] "Generic (PLEG): container finished" podID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerID="2479cd718c09c05299b3e1c074ce1bed56d62b1452e0f3431abdd454feb69b80" exitCode=0 Sep 10 12:11:43.968612 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:43.968202 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" event={"ID":"f16f2c7e-7fc6-4159-b421-f1398514070b","Type":"ContainerDied","Data":"2479cd718c09c05299b3e1c074ce1bed56d62b1452e0f3431abdd454feb69b80"} Sep 10 12:11:43.968612 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:43.968609 2582 scope.go:117] "RemoveContainer" containerID="2479cd718c09c05299b3e1c074ce1bed56d62b1452e0f3431abdd454feb69b80" Sep 10 12:11:43.969709 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:43.969681 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:11:43.969775 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:43.969760 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:11:43.969817 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:43.969796 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:11:44.975633 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:44.975597 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" event={"ID":"f16f2c7e-7fc6-4159-b421-f1398514070b","Type":"ContainerStarted","Data":"1d7f038e6603dbc704ead2ce9b17a5473b92d9ed29b96e535bc5427936106bed"} Sep 10 12:11:45.315959 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:45.315854 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:45.316168 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:45.316123 2582 patch_prober.go:28] interesting pod/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 12:11:45.316251 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:45.316205 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" podUID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 12:11:46.316101 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:46.316062 2582 patch_prober.go:28] interesting pod/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 12:11:46.316709 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:46.316680 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" podUID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 12:11:47.320494 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:47.320463 2582 patch_prober.go:28] interesting pod/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 12:11:47.320950 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:47.320908 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" podUID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 12:11:48.316595 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:48.316549 2582 patch_prober.go:28] interesting pod/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 12:11:48.316800 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:48.316629 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" podUID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 12:11:49.315941 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:49.315906 2582 patch_prober.go:28] interesting pod/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 12:11:49.316573 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:49.316540 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" podUID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 12:11:50.320040 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:50.320013 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:50.320505 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:50.320483 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:50.321097 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:50.321078 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:11:54.260106 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:11:54.260068 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:12:11.465410 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:11.465366 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:12:24.659130 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.658936 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l"] Sep 10 12:12:24.663293 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.663273 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.666221 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.666203 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ed5ac\"/\"openshift-service-ca.crt\"" Sep 10 12:12:24.667382 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.667361 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ed5ac\"/\"kube-root-ca.crt\"" Sep 10 12:12:24.667506 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.667481 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ed5ac\"/\"auth-test-model-7ed5ac-kserve-self-signed-certs\"" Sep 10 12:12:24.667578 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.667520 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ed5ac\"/\"default-dockercfg-69mpl\"" Sep 10 12:12:24.673443 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.673422 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l"] Sep 10 12:12:24.772819 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.772785 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-tls-certs\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.772819 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.772824 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-tmp-dir\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.773073 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.772856 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-home\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.773073 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.772872 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-dshm\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.773073 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.772945 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkgjg\" (UniqueName: \"kubernetes.io/projected/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-kube-api-access-gkgjg\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.773073 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.773065 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-model-cache\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.874011 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.873976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-tls-certs\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.874212 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.874017 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-tmp-dir\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.874212 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.874165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-home\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.874212 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.874204 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-dshm\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.874381 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.874232 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkgjg\" (UniqueName: \"kubernetes.io/projected/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-kube-api-access-gkgjg\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.874381 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.874311 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-model-cache\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.874496 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.874413 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-tmp-dir\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.874496 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.874477 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-home\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.874632 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.874616 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-model-cache\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.876594 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.876563 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-dshm\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.876745 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.876727 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-tls-certs\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.882629 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.882604 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkgjg\" (UniqueName: \"kubernetes.io/projected/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-kube-api-access-gkgjg\") pod \"auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:24.973860 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:24.973831 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:25.260184 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:25.260160 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l"] Sep 10 12:12:25.261877 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:12:25.261841 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3e206c3_1c8b_47a2_b2ba_03e414a8e8cf.slice/crio-0e7e2485d72149de3803361bc6d04966d675e814d3faa4a727294f9e3b78dd4a WatchSource:0}: Error finding container 0e7e2485d72149de3803361bc6d04966d675e814d3faa4a727294f9e3b78dd4a: Status 404 returned error can't find the container with id 0e7e2485d72149de3803361bc6d04966d675e814d3faa4a727294f9e3b78dd4a Sep 10 12:12:26.230027 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:26.229992 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" event={"ID":"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf","Type":"ContainerStarted","Data":"78d1fc18f0162117ad21d2b99389667721c8b06cfcfb5df9e32ea153a624a8ed"} Sep 10 12:12:26.230423 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:26.230040 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" event={"ID":"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf","Type":"ContainerStarted","Data":"0e7e2485d72149de3803361bc6d04966d675e814d3faa4a727294f9e3b78dd4a"} Sep 10 12:12:26.248704 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:26.248664 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" podStartSLOduration=2.24864976 podStartE2EDuration="2.24864976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:12:26.247952423 +0000 UTC m=+1254.127341856" watchObservedRunningTime="2026-09-10 12:12:26.24864976 +0000 UTC m=+1254.128039192" Sep 10 12:12:28.670244 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:28.670210 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:12:32.767538 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:32.767504 2582 scope.go:117] "RemoveContainer" containerID="e3a2149ad972c3a2f7b131240ae4ff81b472b16c8aa31e6122fc2dd9d37ea05b" Sep 10 12:12:34.974039 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:34.974000 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:34.974455 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:34.974059 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:34.986442 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:34.986418 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:35.299800 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:35.299703 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:12:56.353463 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:12:56.353434 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:13:00.673827 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:00.673793 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8"] Sep 10 12:13:00.677312 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:00.677295 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8" Sep 10 12:13:00.680122 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:00.680096 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:13:00.680241 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:00.680126 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:13:00.685707 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:00.685685 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8"] Sep 10 12:13:00.715592 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:00.715560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53477df0-168a-4646-b02f-f67f69a4fa00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8\" (UID: \"53477df0-168a-4646-b02f-f67f69a4fa00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8" Sep 10 12:13:00.816617 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:00.816580 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53477df0-168a-4646-b02f-f67f69a4fa00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8\" (UID: \"53477df0-168a-4646-b02f-f67f69a4fa00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8" Sep 10 12:13:00.817206 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:00.817184 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53477df0-168a-4646-b02f-f67f69a4fa00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8\" (UID: \"53477df0-168a-4646-b02f-f67f69a4fa00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8" Sep 10 12:13:00.988818 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:00.988785 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8" Sep 10 12:13:01.259842 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:01.259816 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8"] Sep 10 12:13:01.261907 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:13:01.261882 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53477df0_168a_4646_b02f_f67f69a4fa00.slice/crio-aa265399475f644cd2dcbe7e9e00abdcb840c9c9109cbfa35de2a4aaffdd9983 WatchSource:0}: Error finding container aa265399475f644cd2dcbe7e9e00abdcb840c9c9109cbfa35de2a4aaffdd9983: Status 404 returned error can't find the container with id aa265399475f644cd2dcbe7e9e00abdcb840c9c9109cbfa35de2a4aaffdd9983 Sep 10 12:13:01.436554 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:01.436520 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8" event={"ID":"53477df0-168a-4646-b02f-f67f69a4fa00","Type":"ContainerStarted","Data":"aa265399475f644cd2dcbe7e9e00abdcb840c9c9109cbfa35de2a4aaffdd9983"} Sep 10 12:13:02.444077 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:02.444038 2582 generic.go:356] "Generic (PLEG): container finished" podID="53477df0-168a-4646-b02f-f67f69a4fa00" containerID="aa557d132ebdb925547bbf1353bfe244bd3ce195ded340e896b5298cd0eeb94e" exitCode=0 Sep 10 12:13:02.444484 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:02.444091 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8" event={"ID":"53477df0-168a-4646-b02f-f67f69a4fa00","Type":"ContainerDied","Data":"aa557d132ebdb925547bbf1353bfe244bd3ce195ded340e896b5298cd0eeb94e"} Sep 10 12:13:03.587596 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:03.587572 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8" Sep 10 12:13:03.642599 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:03.642570 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53477df0-168a-4646-b02f-f67f69a4fa00-maas-api-service-ca\") pod \"53477df0-168a-4646-b02f-f67f69a4fa00\" (UID: \"53477df0-168a-4646-b02f-f67f69a4fa00\") " Sep 10 12:13:03.642951 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:03.642927 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53477df0-168a-4646-b02f-f67f69a4fa00-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53477df0-168a-4646-b02f-f67f69a4fa00" (UID: "53477df0-168a-4646-b02f-f67f69a4fa00"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:13:03.743993 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:03.743911 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53477df0-168a-4646-b02f-f67f69a4fa00-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:13:04.456277 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:04.456249 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8" Sep 10 12:13:04.456446 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:04.456288 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8" event={"ID":"53477df0-168a-4646-b02f-f67f69a4fa00","Type":"ContainerDied","Data":"aa265399475f644cd2dcbe7e9e00abdcb840c9c9109cbfa35de2a4aaffdd9983"} Sep 10 12:13:04.456446 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:04.456329 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa265399475f644cd2dcbe7e9e00abdcb840c9c9109cbfa35de2a4aaffdd9983" Sep 10 12:13:15.951169 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:15.951108 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:13:16.478547 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:16.478508 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8"] Sep 10 12:13:16.482268 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:16.482243 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e74b3f66-g6xk8"] Sep 10 12:13:16.629903 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:16.629872 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53477df0-168a-4646-b02f-f67f69a4fa00" path="/var/lib/kubelet/pods/53477df0-168a-4646-b02f-f67f69a4fa00/volumes" Sep 10 12:13:32.789363 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:32.789335 2582 scope.go:117] "RemoveContainer" containerID="44eadfa1d342466102aeff9dbb5b7121490db9335157e1d78e132e55f0032ebe" Sep 10 12:13:32.799855 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:32.799830 2582 scope.go:117] "RemoveContainer" containerID="de61d48129bae198ef503d498207d832cb97322daa6e4c8374249968c6c113fb" Sep 10 12:13:32.807956 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:32.807936 2582 scope.go:117] "RemoveContainer" containerID="c59d14f79838564931bdf4b4863b2868e92d6caf24c13092618dbf3114b211ca" Sep 10 12:13:33.657406 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:33.657374 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:13:49.671029 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:13:49.670987 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:14:05.557267 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:05.557187 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:14:14.954842 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:14.954806 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk"] Sep 10 12:14:14.955324 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:14.955076 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" podUID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerName="istio-proxy" containerID="cri-o://1d7f038e6603dbc704ead2ce9b17a5473b92d9ed29b96e535bc5427936106bed" gracePeriod=30 Sep 10 12:14:20.213798 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.213770 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:14:20.292057 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.291974 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-workload-certs\") pod \"f16f2c7e-7fc6-4159-b421-f1398514070b\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " Sep 10 12:14:20.292057 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292011 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-token\") pod \"f16f2c7e-7fc6-4159-b421-f1398514070b\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " Sep 10 12:14:20.292057 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292029 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-data\") pod \"f16f2c7e-7fc6-4159-b421-f1398514070b\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " Sep 10 12:14:20.292348 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292087 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-workload-socket\") pod \"f16f2c7e-7fc6-4159-b421-f1398514070b\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " Sep 10 12:14:20.292348 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292134 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-credential-socket\") pod \"f16f2c7e-7fc6-4159-b421-f1398514070b\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " Sep 10 12:14:20.292348 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292177 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd72h\" (UniqueName: \"kubernetes.io/projected/f16f2c7e-7fc6-4159-b421-f1398514070b-kube-api-access-dd72h\") pod \"f16f2c7e-7fc6-4159-b421-f1398514070b\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " Sep 10 12:14:20.292348 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292202 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f2c7e-7fc6-4159-b421-f1398514070b-istiod-ca-cert\") pod \"f16f2c7e-7fc6-4159-b421-f1398514070b\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " Sep 10 12:14:20.292348 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292271 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f16f2c7e-7fc6-4159-b421-f1398514070b" (UID: "f16f2c7e-7fc6-4159-b421-f1398514070b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:14:20.292348 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292280 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-envoy\") pod \"f16f2c7e-7fc6-4159-b421-f1398514070b\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " Sep 10 12:14:20.292666 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292384 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-podinfo\") pod \"f16f2c7e-7fc6-4159-b421-f1398514070b\" (UID: \"f16f2c7e-7fc6-4159-b421-f1398514070b\") " Sep 10 12:14:20.292666 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292425 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-data" (OuterVolumeSpecName: "istio-data") pod "f16f2c7e-7fc6-4159-b421-f1398514070b" (UID: "f16f2c7e-7fc6-4159-b421-f1398514070b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:14:20.292766 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292662 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f16f2c7e-7fc6-4159-b421-f1398514070b" (UID: "f16f2c7e-7fc6-4159-b421-f1398514070b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:14:20.292766 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292675 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f16f2c7e-7fc6-4159-b421-f1398514070b" (UID: "f16f2c7e-7fc6-4159-b421-f1398514070b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:14:20.292766 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292684 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16f2c7e-7fc6-4159-b421-f1398514070b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f16f2c7e-7fc6-4159-b421-f1398514070b" (UID: "f16f2c7e-7fc6-4159-b421-f1398514070b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:14:20.293091 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292806 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:14:20.293091 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292826 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:14:20.293091 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292839 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:14:20.293091 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292853 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:14:20.293091 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.292867 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f2c7e-7fc6-4159-b421-f1398514070b-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:14:20.294682 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.294619 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f16f2c7e-7fc6-4159-b421-f1398514070b" (UID: "f16f2c7e-7fc6-4159-b421-f1398514070b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:14:20.294816 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.294697 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f16f2c7e-7fc6-4159-b421-f1398514070b" (UID: "f16f2c7e-7fc6-4159-b421-f1398514070b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:14:20.294816 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.294722 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f16f2c7e-7fc6-4159-b421-f1398514070b" (UID: "f16f2c7e-7fc6-4159-b421-f1398514070b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:14:20.294816 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.294752 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16f2c7e-7fc6-4159-b421-f1398514070b-kube-api-access-dd72h" (OuterVolumeSpecName: "kube-api-access-dd72h") pod "f16f2c7e-7fc6-4159-b421-f1398514070b" (UID: "f16f2c7e-7fc6-4159-b421-f1398514070b"). InnerVolumeSpecName "kube-api-access-dd72h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:14:20.393566 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.393537 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:14:20.393566 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.393563 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:14:20.393566 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.393572 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16f2c7e-7fc6-4159-b421-f1398514070b-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:14:20.393797 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.393580 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd72h\" (UniqueName: \"kubernetes.io/projected/f16f2c7e-7fc6-4159-b421-f1398514070b-kube-api-access-dd72h\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:14:20.895111 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.895014 2582 generic.go:356] "Generic (PLEG): container finished" podID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerID="1d7f038e6603dbc704ead2ce9b17a5473b92d9ed29b96e535bc5427936106bed" exitCode=0 Sep 10 12:14:20.895111 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.895087 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" event={"ID":"f16f2c7e-7fc6-4159-b421-f1398514070b","Type":"ContainerDied","Data":"1d7f038e6603dbc704ead2ce9b17a5473b92d9ed29b96e535bc5427936106bed"} Sep 10 12:14:20.895328 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.895120 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" Sep 10 12:14:20.895328 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.895166 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk" event={"ID":"f16f2c7e-7fc6-4159-b421-f1398514070b","Type":"ContainerDied","Data":"f703901740cb696e0b6d705197134a40c4a2ba4a83c5a46b45f53456d81f8e1c"} Sep 10 12:14:20.895328 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.895198 2582 scope.go:117] "RemoveContainer" containerID="1d7f038e6603dbc704ead2ce9b17a5473b92d9ed29b96e535bc5427936106bed" Sep 10 12:14:20.906166 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.906133 2582 scope.go:117] "RemoveContainer" containerID="2479cd718c09c05299b3e1c074ce1bed56d62b1452e0f3431abdd454feb69b80" Sep 10 12:14:20.914699 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.914681 2582 scope.go:117] "RemoveContainer" containerID="1d7f038e6603dbc704ead2ce9b17a5473b92d9ed29b96e535bc5427936106bed" Sep 10 12:14:20.914962 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:14:20.914943 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7f038e6603dbc704ead2ce9b17a5473b92d9ed29b96e535bc5427936106bed\": container with ID starting with 1d7f038e6603dbc704ead2ce9b17a5473b92d9ed29b96e535bc5427936106bed not found: ID does not exist" containerID="1d7f038e6603dbc704ead2ce9b17a5473b92d9ed29b96e535bc5427936106bed" Sep 10 12:14:20.915016 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.914971 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7f038e6603dbc704ead2ce9b17a5473b92d9ed29b96e535bc5427936106bed"} err="failed to get container status \"1d7f038e6603dbc704ead2ce9b17a5473b92d9ed29b96e535bc5427936106bed\": rpc error: code = NotFound desc = could not find container \"1d7f038e6603dbc704ead2ce9b17a5473b92d9ed29b96e535bc5427936106bed\": container with ID starting with 1d7f038e6603dbc704ead2ce9b17a5473b92d9ed29b96e535bc5427936106bed not found: ID does not exist" Sep 10 12:14:20.915016 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.914987 2582 scope.go:117] "RemoveContainer" containerID="2479cd718c09c05299b3e1c074ce1bed56d62b1452e0f3431abdd454feb69b80" Sep 10 12:14:20.915295 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:14:20.915276 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2479cd718c09c05299b3e1c074ce1bed56d62b1452e0f3431abdd454feb69b80\": container with ID starting with 2479cd718c09c05299b3e1c074ce1bed56d62b1452e0f3431abdd454feb69b80 not found: ID does not exist" containerID="2479cd718c09c05299b3e1c074ce1bed56d62b1452e0f3431abdd454feb69b80" Sep 10 12:14:20.915346 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.915300 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2479cd718c09c05299b3e1c074ce1bed56d62b1452e0f3431abdd454feb69b80"} err="failed to get container status \"2479cd718c09c05299b3e1c074ce1bed56d62b1452e0f3431abdd454feb69b80\": rpc error: code = NotFound desc = could not find container \"2479cd718c09c05299b3e1c074ce1bed56d62b1452e0f3431abdd454feb69b80\": container with ID starting with 2479cd718c09c05299b3e1c074ce1bed56d62b1452e0f3431abdd454feb69b80 not found: ID does not exist" Sep 10 12:14:20.915815 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.915797 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk"] Sep 10 12:14:20.918795 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:20.918775 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e74b3f66-openshift-default-545997585b-9zbdk"] Sep 10 12:14:21.457779 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:21.457746 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:14:22.634137 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:22.634090 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f16f2c7e-7fc6-4159-b421-f1398514070b" path="/var/lib/kubelet/pods/f16f2c7e-7fc6-4159-b421-f1398514070b/volumes" Sep 10 12:14:38.678280 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:38.678247 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:14:40.795217 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.795184 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7961-b321d8"] Sep 10 12:14:40.795584 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.795571 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerName="istio-proxy" Sep 10 12:14:40.795632 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.795585 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerName="istio-proxy" Sep 10 12:14:40.795632 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.795598 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerName="istio-proxy" Sep 10 12:14:40.795632 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.795603 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerName="istio-proxy" Sep 10 12:14:40.795632 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.795615 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53477df0-168a-4646-b02f-f67f69a4fa00" containerName="revoke-keys" Sep 10 12:14:40.795632 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.795620 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="53477df0-168a-4646-b02f-f67f69a4fa00" containerName="revoke-keys" Sep 10 12:14:40.795784 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.795689 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerName="istio-proxy" Sep 10 12:14:40.795784 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.795697 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="53477df0-168a-4646-b02f-f67f69a4fa00" containerName="revoke-keys" Sep 10 12:14:40.795784 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.795704 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f16f2c7e-7fc6-4159-b421-f1398514070b" containerName="istio-proxy" Sep 10 12:14:40.800087 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.800070 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7961-b321d8" Sep 10 12:14:40.803960 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.803938 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7cc6f\"" Sep 10 12:14:40.813131 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.813109 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7961-b321d8"] Sep 10 12:14:40.892849 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.892810 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xpdc\" (UniqueName: \"kubernetes.io/projected/49120d3d-8f18-4ecd-8a0c-f5580044009a-kube-api-access-4xpdc\") pod \"mt-curl-7961-b321d8\" (UID: \"49120d3d-8f18-4ecd-8a0c-f5580044009a\") " pod="opendatahub/mt-curl-7961-b321d8" Sep 10 12:14:40.993369 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:40.993335 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpdc\" (UniqueName: \"kubernetes.io/projected/49120d3d-8f18-4ecd-8a0c-f5580044009a-kube-api-access-4xpdc\") pod \"mt-curl-7961-b321d8\" (UID: \"49120d3d-8f18-4ecd-8a0c-f5580044009a\") " pod="opendatahub/mt-curl-7961-b321d8" Sep 10 12:14:41.007064 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:41.007031 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xpdc\" (UniqueName: \"kubernetes.io/projected/49120d3d-8f18-4ecd-8a0c-f5580044009a-kube-api-access-4xpdc\") pod \"mt-curl-7961-b321d8\" (UID: \"49120d3d-8f18-4ecd-8a0c-f5580044009a\") " pod="opendatahub/mt-curl-7961-b321d8" Sep 10 12:14:41.110031 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:41.109948 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7961-b321d8" Sep 10 12:14:41.387125 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:41.387036 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7961-b321d8"] Sep 10 12:14:41.390563 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:14:41.390514 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49120d3d_8f18_4ecd_8a0c_f5580044009a.slice/crio-63c4df4d5fa649882425f181bd2f5dbe5b608fa23ac9c9c5b512143dba3d7a30 WatchSource:0}: Error finding container 63c4df4d5fa649882425f181bd2f5dbe5b608fa23ac9c9c5b512143dba3d7a30: Status 404 returned error can't find the container with id 63c4df4d5fa649882425f181bd2f5dbe5b608fa23ac9c9c5b512143dba3d7a30 Sep 10 12:14:42.024511 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:42.024460 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7961-b321d8" event={"ID":"49120d3d-8f18-4ecd-8a0c-f5580044009a","Type":"ContainerStarted","Data":"63c4df4d5fa649882425f181bd2f5dbe5b608fa23ac9c9c5b512143dba3d7a30"} Sep 10 12:14:43.031256 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:43.031227 2582 generic.go:356] "Generic (PLEG): container finished" podID="49120d3d-8f18-4ecd-8a0c-f5580044009a" containerID="6d2f86071a5d06f33444bbb47fa0c738ca501c639c6df927e1f8c17da9955ef0" exitCode=0 Sep 10 12:14:43.031646 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:43.031314 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7961-b321d8" event={"ID":"49120d3d-8f18-4ecd-8a0c-f5580044009a","Type":"ContainerDied","Data":"6d2f86071a5d06f33444bbb47fa0c738ca501c639c6df927e1f8c17da9955ef0"} Sep 10 12:14:44.172325 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:44.172301 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7961-b321d8" Sep 10 12:14:44.223354 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:44.223329 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xpdc\" (UniqueName: \"kubernetes.io/projected/49120d3d-8f18-4ecd-8a0c-f5580044009a-kube-api-access-4xpdc\") pod \"49120d3d-8f18-4ecd-8a0c-f5580044009a\" (UID: \"49120d3d-8f18-4ecd-8a0c-f5580044009a\") " Sep 10 12:14:44.225747 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:44.225715 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49120d3d-8f18-4ecd-8a0c-f5580044009a-kube-api-access-4xpdc" (OuterVolumeSpecName: "kube-api-access-4xpdc") pod "49120d3d-8f18-4ecd-8a0c-f5580044009a" (UID: "49120d3d-8f18-4ecd-8a0c-f5580044009a"). InnerVolumeSpecName "kube-api-access-4xpdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:14:44.228589 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:44.228568 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7961-b321d8_49120d3d-8f18-4ecd-8a0c-f5580044009a/mt-curl-7961-b321d8/0.log" Sep 10 12:14:44.263623 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:44.263601 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7961-b321d8"] Sep 10 12:14:44.270935 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:44.270914 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7961-b321d8"] Sep 10 12:14:44.324218 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:44.324118 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xpdc\" (UniqueName: \"kubernetes.io/projected/49120d3d-8f18-4ecd-8a0c-f5580044009a-kube-api-access-4xpdc\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:14:44.631392 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:44.631307 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49120d3d-8f18-4ecd-8a0c-f5580044009a" path="/var/lib/kubelet/pods/49120d3d-8f18-4ecd-8a0c-f5580044009a/volumes" Sep 10 12:14:45.043440 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:45.043413 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7961-b321d8" Sep 10 12:14:45.043621 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:45.043467 2582 scope.go:117] "RemoveContainer" containerID="6d2f86071a5d06f33444bbb47fa0c738ca501c639c6df927e1f8c17da9955ef0" Sep 10 12:14:46.872344 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:46.872312 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l"] Sep 10 12:14:50.465510 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:14:50.465473 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:15:00.161037 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.161000 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7"] Sep 10 12:15:00.161490 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.161473 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49120d3d-8f18-4ecd-8a0c-f5580044009a" containerName="mt-curl-7961-b321d8" Sep 10 12:15:00.161535 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.161494 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="49120d3d-8f18-4ecd-8a0c-f5580044009a" containerName="mt-curl-7961-b321d8" Sep 10 12:15:00.161579 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.161568 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="49120d3d-8f18-4ecd-8a0c-f5580044009a" containerName="mt-curl-7961-b321d8" Sep 10 12:15:00.165991 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.165974 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817375-bv82x"] Sep 10 12:15:00.166119 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.166100 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7" Sep 10 12:15:00.169336 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.169314 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn"] Sep 10 12:15:00.169463 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.169409 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817375-bv82x" Sep 10 12:15:00.169463 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.169413 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:15:00.172844 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.172795 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94"] Sep 10 12:15:00.172956 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.172868 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn" Sep 10 12:15:00.176731 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.176712 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817375-bv82x"] Sep 10 12:15:00.176883 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.176807 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94" Sep 10 12:15:00.181237 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.181218 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7"] Sep 10 12:15:00.181871 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.181851 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7" Sep 10 12:15:00.187089 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.186057 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn"] Sep 10 12:15:00.188198 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.188178 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94"] Sep 10 12:15:00.193479 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.193460 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817375-bv82x" Sep 10 12:15:00.238579 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.238527 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn" Sep 10 12:15:00.252104 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.251903 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94" Sep 10 12:15:00.470048 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.470027 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7"] Sep 10 12:15:00.474899 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:15:00.474868 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1575f8fb_3d05_4ada_9b88_df5698947808.slice/crio-0fde3eefe551718eea45ed1d0a2a9adf866b596fc97e4acc40ae21238b40f832 WatchSource:0}: Error finding container 0fde3eefe551718eea45ed1d0a2a9adf866b596fc97e4acc40ae21238b40f832: Status 404 returned error can't find the container with id 0fde3eefe551718eea45ed1d0a2a9adf866b596fc97e4acc40ae21238b40f832 Sep 10 12:15:00.487255 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.487234 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817375-bv82x"] Sep 10 12:15:00.489403 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:15:00.489382 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd640259_5600_4652_8b01_c0c6e9dab37c.slice/crio-817fb10d202571771580db8d26fcceb358e5fcbfb41022e208062ecbce674a48 WatchSource:0}: Error finding container 817fb10d202571771580db8d26fcceb358e5fcbfb41022e208062ecbce674a48: Status 404 returned error can't find the container with id 817fb10d202571771580db8d26fcceb358e5fcbfb41022e208062ecbce674a48 Sep 10 12:15:00.546004 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.545975 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn"] Sep 10 12:15:00.547443 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:15:00.547423 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163819d9_5ad9_4076_9b79_453a90857a39.slice/crio-30379435abd2237e6f0f18beb7da0f139a66c74d756bba90592edcc136c21398 WatchSource:0}: Error finding container 30379435abd2237e6f0f18beb7da0f139a66c74d756bba90592edcc136c21398: Status 404 returned error can't find the container with id 30379435abd2237e6f0f18beb7da0f139a66c74d756bba90592edcc136c21398 Sep 10 12:15:00.563865 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:00.563846 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94"] Sep 10 12:15:00.565597 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:15:00.565573 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf4c770_1f43_4eb6_8699_13e0463b30bd.slice/crio-7081b9906fe6258b353e4d8fc67e1395f7603b092a4a864d23c4060912721326 WatchSource:0}: Error finding container 7081b9906fe6258b353e4d8fc67e1395f7603b092a4a864d23c4060912721326: Status 404 returned error can't find the container with id 7081b9906fe6258b353e4d8fc67e1395f7603b092a4a864d23c4060912721326 Sep 10 12:15:01.138531 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:01.138427 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817375-bv82x" event={"ID":"dd640259-5600-4652-8b01-c0c6e9dab37c","Type":"ContainerStarted","Data":"43919cc5083635ed457ca1a31a3129ca461813429354fe71bab608200e75750e"} Sep 10 12:15:01.138531 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:01.138486 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817375-bv82x" event={"ID":"dd640259-5600-4652-8b01-c0c6e9dab37c","Type":"ContainerStarted","Data":"817fb10d202571771580db8d26fcceb358e5fcbfb41022e208062ecbce674a48"} Sep 10 12:15:01.140013 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:01.139987 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94" event={"ID":"eaf4c770-1f43-4eb6-8699-13e0463b30bd","Type":"ContainerStarted","Data":"a31e2de93e756a84aacf0932fe647620038a453653dd9c8cade929847deaed26"} Sep 10 12:15:01.140163 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:01.140033 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94" event={"ID":"eaf4c770-1f43-4eb6-8699-13e0463b30bd","Type":"ContainerStarted","Data":"7081b9906fe6258b353e4d8fc67e1395f7603b092a4a864d23c4060912721326"} Sep 10 12:15:01.141535 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:01.141509 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7" event={"ID":"1575f8fb-3d05-4ada-9b88-df5698947808","Type":"ContainerStarted","Data":"7a3ff55a3a7800ec0af812f146ad6718afb1e7ba31cbbd73aa7babcf3e5d3bba"} Sep 10 12:15:01.141653 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:01.141549 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7" event={"ID":"1575f8fb-3d05-4ada-9b88-df5698947808","Type":"ContainerStarted","Data":"0fde3eefe551718eea45ed1d0a2a9adf866b596fc97e4acc40ae21238b40f832"} Sep 10 12:15:01.143082 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:01.143059 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn" event={"ID":"163819d9-5ad9-4076-9b79-453a90857a39","Type":"ContainerStarted","Data":"d378c9dda53a652b686664fa04ede5f2d81db6b99aa8862a456c8b2f2f063e04"} Sep 10 12:15:01.143227 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:01.143098 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn" event={"ID":"163819d9-5ad9-4076-9b79-453a90857a39","Type":"ContainerStarted","Data":"30379435abd2237e6f0f18beb7da0f139a66c74d756bba90592edcc136c21398"} Sep 10 12:15:01.161955 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:01.161888 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817375-bv82x" podStartSLOduration=1.16187063 podStartE2EDuration="1.16187063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:15:01.160285341 +0000 UTC m=+1409.039674774" watchObservedRunningTime="2026-09-10 12:15:01.16187063 +0000 UTC m=+1409.041260066" Sep 10 12:15:01.185113 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:01.185067 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94" podStartSLOduration=1.185054108 podStartE2EDuration="1.185054108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:15:01.182594386 +0000 UTC m=+1409.061983818" watchObservedRunningTime="2026-09-10 12:15:01.185054108 +0000 UTC m=+1409.064443532" Sep 10 12:15:01.206015 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:01.205963 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn" podStartSLOduration=1.20594691 podStartE2EDuration="1.20594691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:15:01.204766864 +0000 UTC m=+1409.084156295" watchObservedRunningTime="2026-09-10 12:15:01.20594691 +0000 UTC m=+1409.085336342" Sep 10 12:15:01.229292 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:01.229238 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7" podStartSLOduration=1.22922293 podStartE2EDuration="1.22922293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:15:01.226278413 +0000 UTC m=+1409.105667844" watchObservedRunningTime="2026-09-10 12:15:01.22922293 +0000 UTC m=+1409.108612361" Sep 10 12:15:01.892233 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:01.892191 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" podUID="d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf" containerName="main" containerID="cri-o://78d1fc18f0162117ad21d2b99389667721c8b06cfcfb5df9e32ea153a624a8ed" gracePeriod=45 Sep 10 12:15:02.143358 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.143330 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:15:02.150278 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.150252 2582 generic.go:356] "Generic (PLEG): container finished" podID="d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf" containerID="78d1fc18f0162117ad21d2b99389667721c8b06cfcfb5df9e32ea153a624a8ed" exitCode=0 Sep 10 12:15:02.150426 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.150289 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" event={"ID":"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf","Type":"ContainerDied","Data":"78d1fc18f0162117ad21d2b99389667721c8b06cfcfb5df9e32ea153a624a8ed"} Sep 10 12:15:02.150426 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.150317 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" Sep 10 12:15:02.150426 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.150330 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l" event={"ID":"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf","Type":"ContainerDied","Data":"0e7e2485d72149de3803361bc6d04966d675e814d3faa4a727294f9e3b78dd4a"} Sep 10 12:15:02.150426 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.150352 2582 scope.go:117] "RemoveContainer" containerID="78d1fc18f0162117ad21d2b99389667721c8b06cfcfb5df9e32ea153a624a8ed" Sep 10 12:15:02.152322 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.152286 2582 generic.go:356] "Generic (PLEG): container finished" podID="163819d9-5ad9-4076-9b79-453a90857a39" containerID="d378c9dda53a652b686664fa04ede5f2d81db6b99aa8862a456c8b2f2f063e04" exitCode=0 Sep 10 12:15:02.152425 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.152369 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn" event={"ID":"163819d9-5ad9-4076-9b79-453a90857a39","Type":"ContainerDied","Data":"d378c9dda53a652b686664fa04ede5f2d81db6b99aa8862a456c8b2f2f063e04"} Sep 10 12:15:02.154116 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.154093 2582 generic.go:356] "Generic (PLEG): container finished" podID="dd640259-5600-4652-8b01-c0c6e9dab37c" containerID="43919cc5083635ed457ca1a31a3129ca461813429354fe71bab608200e75750e" exitCode=0 Sep 10 12:15:02.154221 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.154174 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817375-bv82x" event={"ID":"dd640259-5600-4652-8b01-c0c6e9dab37c","Type":"ContainerDied","Data":"43919cc5083635ed457ca1a31a3129ca461813429354fe71bab608200e75750e"} Sep 10 12:15:02.155861 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.155837 2582 generic.go:356] "Generic (PLEG): container finished" podID="eaf4c770-1f43-4eb6-8699-13e0463b30bd" containerID="a31e2de93e756a84aacf0932fe647620038a453653dd9c8cade929847deaed26" exitCode=0 Sep 10 12:15:02.155958 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.155914 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94" event={"ID":"eaf4c770-1f43-4eb6-8699-13e0463b30bd","Type":"ContainerDied","Data":"a31e2de93e756a84aacf0932fe647620038a453653dd9c8cade929847deaed26"} Sep 10 12:15:02.157439 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.157420 2582 generic.go:356] "Generic (PLEG): container finished" podID="1575f8fb-3d05-4ada-9b88-df5698947808" containerID="7a3ff55a3a7800ec0af812f146ad6718afb1e7ba31cbbd73aa7babcf3e5d3bba" exitCode=0 Sep 10 12:15:02.157527 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.157454 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7" event={"ID":"1575f8fb-3d05-4ada-9b88-df5698947808","Type":"ContainerDied","Data":"7a3ff55a3a7800ec0af812f146ad6718afb1e7ba31cbbd73aa7babcf3e5d3bba"} Sep 10 12:15:02.160824 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.160802 2582 scope.go:117] "RemoveContainer" containerID="78d1fc18f0162117ad21d2b99389667721c8b06cfcfb5df9e32ea153a624a8ed" Sep 10 12:15:02.161098 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:15:02.161069 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d1fc18f0162117ad21d2b99389667721c8b06cfcfb5df9e32ea153a624a8ed\": container with ID starting with 78d1fc18f0162117ad21d2b99389667721c8b06cfcfb5df9e32ea153a624a8ed not found: ID does not exist" containerID="78d1fc18f0162117ad21d2b99389667721c8b06cfcfb5df9e32ea153a624a8ed" Sep 10 12:15:02.161185 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.161104 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d1fc18f0162117ad21d2b99389667721c8b06cfcfb5df9e32ea153a624a8ed"} err="failed to get container status \"78d1fc18f0162117ad21d2b99389667721c8b06cfcfb5df9e32ea153a624a8ed\": rpc error: code = NotFound desc = could not find container \"78d1fc18f0162117ad21d2b99389667721c8b06cfcfb5df9e32ea153a624a8ed\": container with ID starting with 78d1fc18f0162117ad21d2b99389667721c8b06cfcfb5df9e32ea153a624a8ed not found: ID does not exist" Sep 10 12:15:02.203496 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.203466 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-tmp-dir\") pod \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " Sep 10 12:15:02.203496 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.203506 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-home\") pod \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " Sep 10 12:15:02.203945 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.203537 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-tls-certs\") pod \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " Sep 10 12:15:02.203945 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.203577 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-model-cache\") pod \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " Sep 10 12:15:02.203945 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.203692 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkgjg\" (UniqueName: \"kubernetes.io/projected/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-kube-api-access-gkgjg\") pod \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " Sep 10 12:15:02.203945 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.203734 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-dshm\") pod \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\" (UID: \"d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf\") " Sep 10 12:15:02.204347 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.204309 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf" (UID: "d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:15:02.204636 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.204378 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf" (UID: "d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:15:02.204636 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.204460 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-home" (OuterVolumeSpecName: "home") pod "d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf" (UID: "d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:15:02.206407 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.206224 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf" (UID: "d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:15:02.206407 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.206274 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-dshm" (OuterVolumeSpecName: "dshm") pod "d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf" (UID: "d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:15:02.206631 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.206607 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-kube-api-access-gkgjg" (OuterVolumeSpecName: "kube-api-access-gkgjg") pod "d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf" (UID: "d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf"). InnerVolumeSpecName "kube-api-access-gkgjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:15:02.305216 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.305185 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-tmp-dir\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:15:02.305216 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.305214 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-home\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:15:02.305216 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.305223 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-tls-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:15:02.305438 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.305232 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-model-cache\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:15:02.305438 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.305241 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkgjg\" (UniqueName: \"kubernetes.io/projected/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-kube-api-access-gkgjg\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:15:02.305438 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.305248 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf-dshm\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:15:02.473903 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.473878 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l"] Sep 10 12:15:02.479246 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.479221 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/auth-test-model-7ed5ac-kserve-6b88856f67-pxs2l"] Sep 10 12:15:02.629799 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:02.629769 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf" path="/var/lib/kubelet/pods/d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf/volumes" Sep 10 12:15:03.330045 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:03.330016 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7" Sep 10 12:15:03.365520 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:03.365493 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn" Sep 10 12:15:03.367738 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:03.367719 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94" Sep 10 12:15:03.383508 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:03.383488 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817375-bv82x" Sep 10 12:15:04.170851 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:04.170817 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn" event={"ID":"163819d9-5ad9-4076-9b79-453a90857a39","Type":"ContainerDied","Data":"30379435abd2237e6f0f18beb7da0f139a66c74d756bba90592edcc136c21398"} Sep 10 12:15:04.171039 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:04.170867 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30379435abd2237e6f0f18beb7da0f139a66c74d756bba90592edcc136c21398" Sep 10 12:15:04.171039 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:04.170823 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn" Sep 10 12:15:04.172154 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:04.172120 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817375-bv82x" Sep 10 12:15:04.172279 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:04.172183 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817375-bv82x" event={"ID":"dd640259-5600-4652-8b01-c0c6e9dab37c","Type":"ContainerDied","Data":"817fb10d202571771580db8d26fcceb358e5fcbfb41022e208062ecbce674a48"} Sep 10 12:15:04.172279 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:04.172221 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="817fb10d202571771580db8d26fcceb358e5fcbfb41022e208062ecbce674a48" Sep 10 12:15:04.173682 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:04.173652 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94" Sep 10 12:15:04.173840 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:04.173679 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94" event={"ID":"eaf4c770-1f43-4eb6-8699-13e0463b30bd","Type":"ContainerDied","Data":"7081b9906fe6258b353e4d8fc67e1395f7603b092a4a864d23c4060912721326"} Sep 10 12:15:04.173840 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:04.173728 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7081b9906fe6258b353e4d8fc67e1395f7603b092a4a864d23c4060912721326" Sep 10 12:15:04.175078 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:04.175062 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7" Sep 10 12:15:04.175196 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:04.175104 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7" event={"ID":"1575f8fb-3d05-4ada-9b88-df5698947808","Type":"ContainerDied","Data":"0fde3eefe551718eea45ed1d0a2a9adf866b596fc97e4acc40ae21238b40f832"} Sep 10 12:15:04.175196 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:04.175162 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fde3eefe551718eea45ed1d0a2a9adf866b596fc97e4acc40ae21238b40f832" Sep 10 12:15:05.198512 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:05.198477 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:15:10.907129 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907049 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc"] Sep 10 12:15:10.907654 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907633 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1575f8fb-3d05-4ada-9b88-df5698947808" containerName="cleanup" Sep 10 12:15:10.907797 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907657 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1575f8fb-3d05-4ada-9b88-df5698947808" containerName="cleanup" Sep 10 12:15:10.907797 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907672 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf" containerName="main" Sep 10 12:15:10.907797 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907680 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf" containerName="main" Sep 10 12:15:10.907797 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907693 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd640259-5600-4652-8b01-c0c6e9dab37c" containerName="cleanup" Sep 10 12:15:10.907797 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907701 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd640259-5600-4652-8b01-c0c6e9dab37c" containerName="cleanup" Sep 10 12:15:10.907797 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907718 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf4c770-1f43-4eb6-8699-13e0463b30bd" containerName="cleanup" Sep 10 12:15:10.907797 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907727 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf4c770-1f43-4eb6-8699-13e0463b30bd" containerName="cleanup" Sep 10 12:15:10.907797 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907742 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163819d9-5ad9-4076-9b79-453a90857a39" containerName="cleanup" Sep 10 12:15:10.907797 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907750 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="163819d9-5ad9-4076-9b79-453a90857a39" containerName="cleanup" Sep 10 12:15:10.908259 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907853 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaf4c770-1f43-4eb6-8699-13e0463b30bd" containerName="cleanup" Sep 10 12:15:10.908259 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907866 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd640259-5600-4652-8b01-c0c6e9dab37c" containerName="cleanup" Sep 10 12:15:10.908259 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907876 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3e206c3-1c8b-47a2-b2ba-03e414a8e8cf" containerName="main" Sep 10 12:15:10.908259 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907887 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="163819d9-5ad9-4076-9b79-453a90857a39" containerName="cleanup" Sep 10 12:15:10.908259 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.907897 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1575f8fb-3d05-4ada-9b88-df5698947808" containerName="cleanup" Sep 10 12:15:10.911309 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.911289 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:10.913964 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.913933 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-187e7e-openshift-default-dockercfg-p5d6l\"" Sep 10 12:15:10.928064 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:10.928042 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc"] Sep 10 12:15:11.092608 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.092562 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-envoy\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.092765 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.092614 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-credential-socket\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.092765 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.092662 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-token\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.092765 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.092679 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-podinfo\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.092765 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.092706 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istiod-ca-cert\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.092765 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.092726 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-data\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.092955 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.092782 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-workload-socket\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.092955 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.092835 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqmrw\" (UniqueName: \"kubernetes.io/projected/17618a39-7011-4b42-8aeb-5cbeeb925bc9-kube-api-access-zqmrw\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.092955 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.092897 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-workload-certs\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.193949 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.193923 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-envoy\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.194124 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.193957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-credential-socket\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.194124 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.193976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-token\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.194124 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.193994 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-podinfo\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.194124 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.194029 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istiod-ca-cert\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.194124 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.194057 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-data\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.194425 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.194224 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-workload-socket\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.194425 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.194273 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmrw\" (UniqueName: \"kubernetes.io/projected/17618a39-7011-4b42-8aeb-5cbeeb925bc9-kube-api-access-zqmrw\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.194425 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.194349 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-workload-certs\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.194588 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.194446 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-credential-socket\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.194731 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.194686 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-workload-socket\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.194939 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.194919 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-workload-certs\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.195161 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.195127 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-data\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.195228 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.195183 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istiod-ca-cert\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.196737 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.196714 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-envoy\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.196998 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.196976 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-podinfo\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.206200 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.206137 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-token\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.206457 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.206427 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmrw\" (UniqueName: \"kubernetes.io/projected/17618a39-7011-4b42-8aeb-5cbeeb925bc9-kube-api-access-zqmrw\") pod \"e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.222489 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.222467 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:11.507595 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.507562 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc"] Sep 10 12:15:11.508652 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:15:11.508626 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17618a39_7011_4b42_8aeb_5cbeeb925bc9.slice/crio-e983ede3e6e59213ad23ce3056e7e028e334752cebcbb684e5f105a80ca0049f WatchSource:0}: Error finding container e983ede3e6e59213ad23ce3056e7e028e334752cebcbb684e5f105a80ca0049f: Status 404 returned error can't find the container with id e983ede3e6e59213ad23ce3056e7e028e334752cebcbb684e5f105a80ca0049f Sep 10 12:15:11.511087 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.511046 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:15:11.511217 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.511127 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:15:11.511217 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:11.511195 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:15:12.226017 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:12.225977 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" event={"ID":"17618a39-7011-4b42-8aeb-5cbeeb925bc9","Type":"ContainerStarted","Data":"f9d18451de1a4f4b601210e2c817a0e78b0ee639e303da01d1bafa034001446d"} Sep 10 12:15:12.226384 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:12.226039 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" event={"ID":"17618a39-7011-4b42-8aeb-5cbeeb925bc9","Type":"ContainerStarted","Data":"e983ede3e6e59213ad23ce3056e7e028e334752cebcbb684e5f105a80ca0049f"} Sep 10 12:15:12.256896 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:12.256850 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" podStartSLOduration=2.256835974 podStartE2EDuration="2.256835974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:15:12.255052395 +0000 UTC m=+1420.134441851" watchObservedRunningTime="2026-09-10 12:15:12.256835974 +0000 UTC m=+1420.136225466" Sep 10 12:15:13.223375 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:13.223340 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:13.228566 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:13.228539 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:13.232206 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:13.232183 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:13.233048 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:13.233031 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:24.021243 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:24.021130 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:15:32.852518 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:32.852486 2582 scope.go:117] "RemoveContainer" containerID="914bdfa44f4371e193f9dd404411774d4d3c4910bb09e80a747c9bc2cdcb301b" Sep 10 12:15:32.861406 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:32.861387 2582 scope.go:117] "RemoveContainer" containerID="b25c0d28bb201f853b89d40ecff98b525a64c0ba16cbdced98064dc10a4466a6" Sep 10 12:15:34.364017 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:34.363979 2582 generic.go:356] "Generic (PLEG): container finished" podID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerID="f9d18451de1a4f4b601210e2c817a0e78b0ee639e303da01d1bafa034001446d" exitCode=0 Sep 10 12:15:34.364521 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:34.364057 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" event={"ID":"17618a39-7011-4b42-8aeb-5cbeeb925bc9","Type":"ContainerDied","Data":"f9d18451de1a4f4b601210e2c817a0e78b0ee639e303da01d1bafa034001446d"} Sep 10 12:15:34.364521 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:34.364484 2582 scope.go:117] "RemoveContainer" containerID="f9d18451de1a4f4b601210e2c817a0e78b0ee639e303da01d1bafa034001446d" Sep 10 12:15:34.365617 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:34.365589 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:15:34.365680 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:34.365657 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:15:34.365717 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:34.365700 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:15:35.372295 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:35.372260 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" event={"ID":"17618a39-7011-4b42-8aeb-5cbeeb925bc9","Type":"ContainerStarted","Data":"cc8c91a07791276d4b7f0e96d7cd1aa010750a16ba1c6db0199ad301d0ba4004"} Sep 10 12:15:36.222776 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:36.222742 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:36.223103 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:36.223021 2582 patch_prober.go:28] interesting pod/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 10 12:15:36.223250 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:36.223220 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" podUID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 10 12:15:37.223935 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:37.223903 2582 patch_prober.go:28] interesting pod/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 10 12:15:37.224544 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:37.224508 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" podUID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 10 12:15:38.223803 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:38.223771 2582 patch_prober.go:28] interesting pod/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 10 12:15:38.224055 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:38.224028 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" podUID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 10 12:15:39.223936 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:39.223885 2582 patch_prober.go:28] interesting pod/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 10 12:15:39.224166 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:39.223959 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" podUID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 10 12:15:40.118170 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.115409 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg"] Sep 10 12:15:40.122858 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.122032 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.125540 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.125514 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c531cb-openshift-default-dockercfg-zn4fh\"" Sep 10 12:15:40.128325 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.128297 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg"] Sep 10 12:15:40.182540 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.182503 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-credential-socket\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.182540 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.182540 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543d6243-a64a-4935-bc2b-713437ca95b7-istio-token\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.182789 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.182566 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543d6243-a64a-4935-bc2b-713437ca95b7-istiod-ca-cert\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.182789 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.182654 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543d6243-a64a-4935-bc2b-713437ca95b7-istio-podinfo\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.182789 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.182698 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-istio-envoy\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.182923 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.182792 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-workload-certs\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.182923 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.182834 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-workload-socket\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.182923 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.182912 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7kjc\" (UniqueName: \"kubernetes.io/projected/543d6243-a64a-4935-bc2b-713437ca95b7-kube-api-access-r7kjc\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.183073 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.182977 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-istio-data\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.223656 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.223608 2582 patch_prober.go:28] interesting pod/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 10 12:15:40.223871 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.223699 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" podUID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 10 12:15:40.283915 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.283872 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-workload-certs\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.283915 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.283906 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-workload-socket\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.284477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.283938 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7kjc\" (UniqueName: \"kubernetes.io/projected/543d6243-a64a-4935-bc2b-713437ca95b7-kube-api-access-r7kjc\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.284477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.283970 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-istio-data\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.284477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.283990 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-credential-socket\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.284477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.284009 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543d6243-a64a-4935-bc2b-713437ca95b7-istio-token\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.284477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.284043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543d6243-a64a-4935-bc2b-713437ca95b7-istiod-ca-cert\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.284477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.284097 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543d6243-a64a-4935-bc2b-713437ca95b7-istio-podinfo\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.284477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.284127 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-istio-envoy\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.284477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.284375 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-workload-socket\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.284477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.284442 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-workload-certs\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.284477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.284452 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-istio-data\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.285088 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.284598 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-credential-socket\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.285088 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.284862 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543d6243-a64a-4935-bc2b-713437ca95b7-istiod-ca-cert\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.286519 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.286493 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-istio-envoy\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.286758 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.286735 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543d6243-a64a-4935-bc2b-713437ca95b7-istio-podinfo\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.293440 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.293413 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543d6243-a64a-4935-bc2b-713437ca95b7-istio-token\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.293556 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.293511 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7kjc\" (UniqueName: \"kubernetes.io/projected/543d6243-a64a-4935-bc2b-713437ca95b7-kube-api-access-r7kjc\") pod \"e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.437807 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.437764 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:40.733058 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.733037 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg"] Sep 10 12:15:40.735090 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:15:40.735059 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543d6243_a64a_4935_bc2b_713437ca95b7.slice/crio-b06e50244dd4f52acdb9ab5f57fe03b23f53c6d3640f4aa84ef9325014a8c0f9 WatchSource:0}: Error finding container b06e50244dd4f52acdb9ab5f57fe03b23f53c6d3640f4aa84ef9325014a8c0f9: Status 404 returned error can't find the container with id b06e50244dd4f52acdb9ab5f57fe03b23f53c6d3640f4aa84ef9325014a8c0f9 Sep 10 12:15:40.737376 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.737344 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:15:40.737488 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.737421 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:15:40.737488 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:40.737472 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:15:41.228094 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:41.228065 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:41.228377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:41.228359 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:41.229008 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:41.228986 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:15:41.406666 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:41.406632 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" event={"ID":"543d6243-a64a-4935-bc2b-713437ca95b7","Type":"ContainerStarted","Data":"d928df88e5b3d31c67111244a0641c280253e61fb0450e6810f73303ff6fd5d4"} Sep 10 12:15:41.407179 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:41.406689 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" event={"ID":"543d6243-a64a-4935-bc2b-713437ca95b7","Type":"ContainerStarted","Data":"b06e50244dd4f52acdb9ab5f57fe03b23f53c6d3640f4aa84ef9325014a8c0f9"} Sep 10 12:15:41.429808 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:41.429766 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" podStartSLOduration=1.429752952 podStartE2EDuration="1.429752952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:15:41.42907138 +0000 UTC m=+1449.308460815" watchObservedRunningTime="2026-09-10 12:15:41.429752952 +0000 UTC m=+1449.309142384" Sep 10 12:15:41.438195 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:41.438174 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:41.439660 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:41.439634 2582 patch_prober.go:28] interesting pod/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Sep 10 12:15:41.439766 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:41.439677 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Sep 10 12:15:42.442823 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:42.442797 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:43.420219 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:43.420181 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:43.421040 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:43.421021 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:15:45.177245 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:15:45.177209 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:16:06.565577 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:06.565540 2582 generic.go:356] "Generic (PLEG): container finished" podID="543d6243-a64a-4935-bc2b-713437ca95b7" containerID="d928df88e5b3d31c67111244a0641c280253e61fb0450e6810f73303ff6fd5d4" exitCode=0 Sep 10 12:16:06.566116 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:06.565621 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" event={"ID":"543d6243-a64a-4935-bc2b-713437ca95b7","Type":"ContainerDied","Data":"d928df88e5b3d31c67111244a0641c280253e61fb0450e6810f73303ff6fd5d4"} Sep 10 12:16:06.566236 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:06.566184 2582 scope.go:117] "RemoveContainer" containerID="d928df88e5b3d31c67111244a0641c280253e61fb0450e6810f73303ff6fd5d4" Sep 10 12:16:06.567474 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:06.567450 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:16:06.567545 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:06.567515 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:16:06.567588 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:06.567547 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:16:06.758919 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:06.758884 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:16:07.573857 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:07.573824 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" event={"ID":"543d6243-a64a-4935-bc2b-713437ca95b7","Type":"ContainerStarted","Data":"1ee0c85ce76fb2c22c2cb2bfe6b3b9ef04898499142746974f65c6caf2e045cc"} Sep 10 12:16:08.438226 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:08.438186 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:16:08.438442 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:08.438417 2582 patch_prober.go:28] interesting pod/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Sep 10 12:16:08.438519 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:08.438477 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Sep 10 12:16:09.439054 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:09.439014 2582 patch_prober.go:28] interesting pod/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Sep 10 12:16:09.439458 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:09.439081 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Sep 10 12:16:10.438595 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:10.438560 2582 patch_prober.go:28] interesting pod/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Sep 10 12:16:10.438766 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:10.438611 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Sep 10 12:16:11.410598 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:11.410542 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh"] Sep 10 12:16:11.413795 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:11.413772 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh" Sep 10 12:16:11.419360 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:11.418886 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:16:11.419360 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:11.419185 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:16:11.424103 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:11.424079 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh"] Sep 10 12:16:11.438967 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:11.438944 2582 patch_prober.go:28] interesting pod/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Sep 10 12:16:11.439094 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:11.439008 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Sep 10 12:16:11.514167 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:11.504933 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f63de1e-4e30-424d-b4e0-4b076e90314d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh\" (UID: \"4f63de1e-4e30-424d-b4e0-4b076e90314d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh" Sep 10 12:16:11.606256 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:11.606222 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f63de1e-4e30-424d-b4e0-4b076e90314d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh\" (UID: \"4f63de1e-4e30-424d-b4e0-4b076e90314d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh" Sep 10 12:16:11.607215 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:11.607191 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f63de1e-4e30-424d-b4e0-4b076e90314d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh\" (UID: \"4f63de1e-4e30-424d-b4e0-4b076e90314d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh" Sep 10 12:16:11.731133 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:11.731100 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh" Sep 10 12:16:12.004217 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:12.004184 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh"] Sep 10 12:16:12.438484 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:12.438446 2582 patch_prober.go:28] interesting pod/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Sep 10 12:16:12.438886 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:12.438504 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Sep 10 12:16:12.612323 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:12.612282 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh" event={"ID":"4f63de1e-4e30-424d-b4e0-4b076e90314d","Type":"ContainerStarted","Data":"e682c4f0fd716db7834a1a0d9b07870e1ee97752f64e6cf4871f7a1da6b83c78"} Sep 10 12:16:12.612523 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:12.612352 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh" event={"ID":"4f63de1e-4e30-424d-b4e0-4b076e90314d","Type":"ContainerStarted","Data":"7e3e06abf81dbc5db3cf1e7f5eb24cc734f5bde5bcb3fe63652f41ef7b5eddbc"} Sep 10 12:16:12.629693 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:12.629629 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh" podStartSLOduration=1.629611135 podStartE2EDuration="1.629611135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:16:12.62895059 +0000 UTC m=+1480.508340022" watchObservedRunningTime="2026-09-10 12:16:12.629611135 +0000 UTC m=+1480.509000567" Sep 10 12:16:13.420867 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:13.420772 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:16:13.442730 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:13.442700 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:16:13.619305 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:13.619264 2582 generic.go:356] "Generic (PLEG): container finished" podID="4f63de1e-4e30-424d-b4e0-4b076e90314d" containerID="e682c4f0fd716db7834a1a0d9b07870e1ee97752f64e6cf4871f7a1da6b83c78" exitCode=0 Sep 10 12:16:13.619493 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:13.619356 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh" event={"ID":"4f63de1e-4e30-424d-b4e0-4b076e90314d","Type":"ContainerDied","Data":"e682c4f0fd716db7834a1a0d9b07870e1ee97752f64e6cf4871f7a1da6b83c78"} Sep 10 12:16:13.620671 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:13.620654 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:16:14.768263 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:14.768237 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh" Sep 10 12:16:14.844520 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:14.844485 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f63de1e-4e30-424d-b4e0-4b076e90314d-maas-api-service-ca\") pod \"4f63de1e-4e30-424d-b4e0-4b076e90314d\" (UID: \"4f63de1e-4e30-424d-b4e0-4b076e90314d\") " Sep 10 12:16:14.844835 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:14.844812 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f63de1e-4e30-424d-b4e0-4b076e90314d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f63de1e-4e30-424d-b4e0-4b076e90314d" (UID: "4f63de1e-4e30-424d-b4e0-4b076e90314d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:16:14.945963 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:14.945933 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f63de1e-4e30-424d-b4e0-4b076e90314d-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:16:15.632023 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:15.631991 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh" Sep 10 12:16:15.632201 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:15.631989 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh" event={"ID":"4f63de1e-4e30-424d-b4e0-4b076e90314d","Type":"ContainerDied","Data":"7e3e06abf81dbc5db3cf1e7f5eb24cc734f5bde5bcb3fe63652f41ef7b5eddbc"} Sep 10 12:16:15.632201 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:15.632115 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e3e06abf81dbc5db3cf1e7f5eb24cc734f5bde5bcb3fe63652f41ef7b5eddbc" Sep 10 12:16:22.088477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:22.088446 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7961-83b9b6"] Sep 10 12:16:22.088873 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:22.088859 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f63de1e-4e30-424d-b4e0-4b076e90314d" containerName="revoke-keys" Sep 10 12:16:22.088918 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:22.088874 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f63de1e-4e30-424d-b4e0-4b076e90314d" containerName="revoke-keys" Sep 10 12:16:22.088958 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:22.088950 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f63de1e-4e30-424d-b4e0-4b076e90314d" containerName="revoke-keys" Sep 10 12:16:22.090930 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:22.090915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7961-83b9b6" Sep 10 12:16:22.093858 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:22.093830 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7cc6f\"" Sep 10 12:16:22.097650 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:22.097627 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7961-83b9b6"] Sep 10 12:16:22.217747 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:22.217718 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv74l\" (UniqueName: \"kubernetes.io/projected/ded4732b-d0fb-4cf4-8652-f99368c54ecd-kube-api-access-zv74l\") pod \"mt-curl-7961-83b9b6\" (UID: \"ded4732b-d0fb-4cf4-8652-f99368c54ecd\") " pod="opendatahub/mt-curl-7961-83b9b6" Sep 10 12:16:22.318880 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:22.318851 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv74l\" (UniqueName: \"kubernetes.io/projected/ded4732b-d0fb-4cf4-8652-f99368c54ecd-kube-api-access-zv74l\") pod \"mt-curl-7961-83b9b6\" (UID: \"ded4732b-d0fb-4cf4-8652-f99368c54ecd\") " pod="opendatahub/mt-curl-7961-83b9b6" Sep 10 12:16:22.329377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:22.329354 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv74l\" (UniqueName: \"kubernetes.io/projected/ded4732b-d0fb-4cf4-8652-f99368c54ecd-kube-api-access-zv74l\") pod \"mt-curl-7961-83b9b6\" (UID: \"ded4732b-d0fb-4cf4-8652-f99368c54ecd\") " pod="opendatahub/mt-curl-7961-83b9b6" Sep 10 12:16:22.403320 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:22.403244 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7961-83b9b6" Sep 10 12:16:22.676203 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:22.676176 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7961-83b9b6"] Sep 10 12:16:22.678271 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:16:22.678246 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podded4732b_d0fb_4cf4_8652_f99368c54ecd.slice/crio-64efb32ee2d1ca16229e4547f42aaf132637d7fdc0537a5a668817ffc37fac29 WatchSource:0}: Error finding container 64efb32ee2d1ca16229e4547f42aaf132637d7fdc0537a5a668817ffc37fac29: Status 404 returned error can't find the container with id 64efb32ee2d1ca16229e4547f42aaf132637d7fdc0537a5a668817ffc37fac29 Sep 10 12:16:22.679855 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:22.679839 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 12:16:23.683735 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:23.683639 2582 generic.go:356] "Generic (PLEG): container finished" podID="ded4732b-d0fb-4cf4-8652-f99368c54ecd" containerID="01cc07bf53e7d8ba337162accb1a0236096f1e7041fc7a9a2dafc586008124d3" exitCode=0 Sep 10 12:16:23.683735 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:23.683715 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7961-83b9b6" event={"ID":"ded4732b-d0fb-4cf4-8652-f99368c54ecd","Type":"ContainerDied","Data":"01cc07bf53e7d8ba337162accb1a0236096f1e7041fc7a9a2dafc586008124d3"} Sep 10 12:16:23.684329 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:23.683752 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7961-83b9b6" event={"ID":"ded4732b-d0fb-4cf4-8652-f99368c54ecd","Type":"ContainerStarted","Data":"64efb32ee2d1ca16229e4547f42aaf132637d7fdc0537a5a668817ffc37fac29"} Sep 10 12:16:24.822356 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:24.822334 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7961-83b9b6" Sep 10 12:16:24.873171 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:24.873127 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7961-83b9b6_ded4732b-d0fb-4cf4-8652-f99368c54ecd/mt-curl-7961-83b9b6/0.log" Sep 10 12:16:24.905585 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:24.905557 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7961-83b9b6"] Sep 10 12:16:24.907631 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:24.907613 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7961-83b9b6"] Sep 10 12:16:24.946871 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:24.946800 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv74l\" (UniqueName: \"kubernetes.io/projected/ded4732b-d0fb-4cf4-8652-f99368c54ecd-kube-api-access-zv74l\") pod \"ded4732b-d0fb-4cf4-8652-f99368c54ecd\" (UID: \"ded4732b-d0fb-4cf4-8652-f99368c54ecd\") " Sep 10 12:16:24.949060 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:24.949028 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded4732b-d0fb-4cf4-8652-f99368c54ecd-kube-api-access-zv74l" (OuterVolumeSpecName: "kube-api-access-zv74l") pod "ded4732b-d0fb-4cf4-8652-f99368c54ecd" (UID: "ded4732b-d0fb-4cf4-8652-f99368c54ecd"). InnerVolumeSpecName "kube-api-access-zv74l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:16:25.047999 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.047964 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv74l\" (UniqueName: \"kubernetes.io/projected/ded4732b-d0fb-4cf4-8652-f99368c54ecd-kube-api-access-zv74l\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:16:25.049031 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.049009 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7961-9e4033"] Sep 10 12:16:25.049448 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.049435 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded4732b-d0fb-4cf4-8652-f99368c54ecd" containerName="mt-curl-7961-83b9b6" Sep 10 12:16:25.049497 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.049450 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded4732b-d0fb-4cf4-8652-f99368c54ecd" containerName="mt-curl-7961-83b9b6" Sep 10 12:16:25.049532 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.049515 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded4732b-d0fb-4cf4-8652-f99368c54ecd" containerName="mt-curl-7961-83b9b6" Sep 10 12:16:25.051799 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.051784 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7961-9e4033" Sep 10 12:16:25.063826 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.063800 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7961-9e4033"] Sep 10 12:16:25.149490 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.149454 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkjzb\" (UniqueName: \"kubernetes.io/projected/7ee6407e-b7e2-4002-86ed-dacaa4994e5b-kube-api-access-rkjzb\") pod \"mt-curl-7961-9e4033\" (UID: \"7ee6407e-b7e2-4002-86ed-dacaa4994e5b\") " pod="opendatahub/mt-curl-7961-9e4033" Sep 10 12:16:25.251020 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.250933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkjzb\" (UniqueName: \"kubernetes.io/projected/7ee6407e-b7e2-4002-86ed-dacaa4994e5b-kube-api-access-rkjzb\") pod \"mt-curl-7961-9e4033\" (UID: \"7ee6407e-b7e2-4002-86ed-dacaa4994e5b\") " pod="opendatahub/mt-curl-7961-9e4033" Sep 10 12:16:25.264821 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.264796 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkjzb\" (UniqueName: \"kubernetes.io/projected/7ee6407e-b7e2-4002-86ed-dacaa4994e5b-kube-api-access-rkjzb\") pod \"mt-curl-7961-9e4033\" (UID: \"7ee6407e-b7e2-4002-86ed-dacaa4994e5b\") " pod="opendatahub/mt-curl-7961-9e4033" Sep 10 12:16:25.360605 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.360569 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7961-9e4033" Sep 10 12:16:25.643463 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.643438 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7961-9e4033"] Sep 10 12:16:25.645194 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:16:25.645166 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ee6407e_b7e2_4002_86ed_dacaa4994e5b.slice/crio-4159e68f31c6cf6472f056803389b29ce809f7cfc63f3e89a577f0db20916b51 WatchSource:0}: Error finding container 4159e68f31c6cf6472f056803389b29ce809f7cfc63f3e89a577f0db20916b51: Status 404 returned error can't find the container with id 4159e68f31c6cf6472f056803389b29ce809f7cfc63f3e89a577f0db20916b51 Sep 10 12:16:25.695756 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.695730 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7961-9e4033" event={"ID":"7ee6407e-b7e2-4002-86ed-dacaa4994e5b","Type":"ContainerStarted","Data":"4159e68f31c6cf6472f056803389b29ce809f7cfc63f3e89a577f0db20916b51"} Sep 10 12:16:25.697227 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.697202 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64efb32ee2d1ca16229e4547f42aaf132637d7fdc0537a5a668817ffc37fac29" Sep 10 12:16:25.697227 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:25.697227 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7961-83b9b6" Sep 10 12:16:25.708452 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:16:25.708422 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7961-83b9b6\" is forbidden: User \"system:node:ip-10-0-136-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-197.ec2.internal' and this object" podUID="ded4732b-d0fb-4cf4-8652-f99368c54ecd" pod="opendatahub/mt-curl-7961-83b9b6" Sep 10 12:16:26.630755 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:26.630722 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ded4732b-d0fb-4cf4-8652-f99368c54ecd" path="/var/lib/kubelet/pods/ded4732b-d0fb-4cf4-8652-f99368c54ecd/volumes" Sep 10 12:16:26.706906 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:26.706874 2582 generic.go:356] "Generic (PLEG): container finished" podID="7ee6407e-b7e2-4002-86ed-dacaa4994e5b" containerID="b8c4372c229f05a23c4f2dc0fb760340bdc79735ff2bde55e3fe49e3170f4144" exitCode=0 Sep 10 12:16:26.707086 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:26.706901 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7961-9e4033" event={"ID":"7ee6407e-b7e2-4002-86ed-dacaa4994e5b","Type":"ContainerDied","Data":"b8c4372c229f05a23c4f2dc0fb760340bdc79735ff2bde55e3fe49e3170f4144"} Sep 10 12:16:27.155931 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:27.155895 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh"] Sep 10 12:16:27.157959 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:27.157936 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-187e7e-vl8sh"] Sep 10 12:16:27.843514 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:27.843492 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7961-9e4033" Sep 10 12:16:27.864918 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:27.864887 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7961-9e4033_7ee6407e-b7e2-4002-86ed-dacaa4994e5b/mt-curl-7961-9e4033/0.log" Sep 10 12:16:27.891786 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:27.891760 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7961-9e4033"] Sep 10 12:16:27.893948 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:27.893929 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7961-9e4033"] Sep 10 12:16:27.976505 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:27.976433 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkjzb\" (UniqueName: \"kubernetes.io/projected/7ee6407e-b7e2-4002-86ed-dacaa4994e5b-kube-api-access-rkjzb\") pod \"7ee6407e-b7e2-4002-86ed-dacaa4994e5b\" (UID: \"7ee6407e-b7e2-4002-86ed-dacaa4994e5b\") " Sep 10 12:16:27.978631 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:27.978603 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee6407e-b7e2-4002-86ed-dacaa4994e5b-kube-api-access-rkjzb" (OuterVolumeSpecName: "kube-api-access-rkjzb") pod "7ee6407e-b7e2-4002-86ed-dacaa4994e5b" (UID: "7ee6407e-b7e2-4002-86ed-dacaa4994e5b"). InnerVolumeSpecName "kube-api-access-rkjzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:16:28.077504 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:28.077469 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkjzb\" (UniqueName: \"kubernetes.io/projected/7ee6407e-b7e2-4002-86ed-dacaa4994e5b-kube-api-access-rkjzb\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:16:28.630978 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:28.630944 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f63de1e-4e30-424d-b4e0-4b076e90314d" path="/var/lib/kubelet/pods/4f63de1e-4e30-424d-b4e0-4b076e90314d/volumes" Sep 10 12:16:28.631320 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:28.631306 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee6407e-b7e2-4002-86ed-dacaa4994e5b" path="/var/lib/kubelet/pods/7ee6407e-b7e2-4002-86ed-dacaa4994e5b/volumes" Sep 10 12:16:28.720639 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:28.720612 2582 scope.go:117] "RemoveContainer" containerID="b8c4372c229f05a23c4f2dc0fb760340bdc79735ff2bde55e3fe49e3170f4144" Sep 10 12:16:28.720639 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:28.720636 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7961-9e4033" Sep 10 12:16:29.369294 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:29.369261 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:16:32.703353 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:32.703316 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 12:16:32.709434 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:32.709415 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 12:16:32.898433 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:32.898396 2582 scope.go:117] "RemoveContainer" containerID="2125c9c1988a1de3f55a0750e5cf39ed18133e68dfc717176fbf2af14b660ae6" Sep 10 12:16:33.637052 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:33.637010 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc"] Sep 10 12:16:33.637277 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:33.637241 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" podUID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerName="istio-proxy" containerID="cri-o://cc8c91a07791276d4b7f0e96d7cd1aa010750a16ba1c6db0199ad301d0ba4004" gracePeriod=30 Sep 10 12:16:33.971890 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:33.971854 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2"] Sep 10 12:16:33.972315 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:33.972302 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee6407e-b7e2-4002-86ed-dacaa4994e5b" containerName="mt-curl-7961-9e4033" Sep 10 12:16:33.972315 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:33.972317 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee6407e-b7e2-4002-86ed-dacaa4994e5b" containerName="mt-curl-7961-9e4033" Sep 10 12:16:33.972398 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:33.972388 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee6407e-b7e2-4002-86ed-dacaa4994e5b" containerName="mt-curl-7961-9e4033" Sep 10 12:16:33.976334 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:33.976319 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2" Sep 10 12:16:33.979203 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:33.979179 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:16:33.979356 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:33.979253 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:16:33.983802 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:33.983773 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2"] Sep 10 12:16:34.032609 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:34.032577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/901df1d7-6141-4b81-9512-772565b69d89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2\" (UID: \"901df1d7-6141-4b81-9512-772565b69d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2" Sep 10 12:16:34.133493 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:34.133462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/901df1d7-6141-4b81-9512-772565b69d89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2\" (UID: \"901df1d7-6141-4b81-9512-772565b69d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2" Sep 10 12:16:34.134178 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:34.134161 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/901df1d7-6141-4b81-9512-772565b69d89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2\" (UID: \"901df1d7-6141-4b81-9512-772565b69d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2" Sep 10 12:16:34.290067 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:34.289976 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2" Sep 10 12:16:34.569714 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:34.569678 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2"] Sep 10 12:16:34.570461 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:16:34.570427 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901df1d7_6141_4b81_9512_772565b69d89.slice/crio-c5591dee226bc423854a4671bc02e765ea97315b6d4504b782752c2e0f704c27 WatchSource:0}: Error finding container c5591dee226bc423854a4671bc02e765ea97315b6d4504b782752c2e0f704c27: Status 404 returned error can't find the container with id c5591dee226bc423854a4671bc02e765ea97315b6d4504b782752c2e0f704c27 Sep 10 12:16:34.757860 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:34.757820 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2" event={"ID":"901df1d7-6141-4b81-9512-772565b69d89","Type":"ContainerStarted","Data":"c5591dee226bc423854a4671bc02e765ea97315b6d4504b782752c2e0f704c27"} Sep 10 12:16:35.770083 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:35.770048 2582 generic.go:356] "Generic (PLEG): container finished" podID="901df1d7-6141-4b81-9512-772565b69d89" containerID="c6be41bd6a779c6a2e8777f1468d2c150899f1e035f07d450f6b3baea38f2129" exitCode=0 Sep 10 12:16:35.770521 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:35.770137 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2" event={"ID":"901df1d7-6141-4b81-9512-772565b69d89","Type":"ContainerDied","Data":"c6be41bd6a779c6a2e8777f1468d2c150899f1e035f07d450f6b3baea38f2129"} Sep 10 12:16:36.942167 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:36.942124 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2" Sep 10 12:16:36.955300 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:36.955275 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/901df1d7-6141-4b81-9512-772565b69d89-maas-api-service-ca\") pod \"901df1d7-6141-4b81-9512-772565b69d89\" (UID: \"901df1d7-6141-4b81-9512-772565b69d89\") " Sep 10 12:16:36.955617 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:36.955594 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901df1d7-6141-4b81-9512-772565b69d89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "901df1d7-6141-4b81-9512-772565b69d89" (UID: "901df1d7-6141-4b81-9512-772565b69d89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:16:37.056512 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:37.056432 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/901df1d7-6141-4b81-9512-772565b69d89-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:16:37.782300 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:37.782266 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2" event={"ID":"901df1d7-6141-4b81-9512-772565b69d89","Type":"ContainerDied","Data":"c5591dee226bc423854a4671bc02e765ea97315b6d4504b782752c2e0f704c27"} Sep 10 12:16:37.782477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:37.782319 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5591dee226bc423854a4671bc02e765ea97315b6d4504b782752c2e0f704c27" Sep 10 12:16:37.782477 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:37.782275 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2" Sep 10 12:16:38.792479 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.792451 2582 generic.go:356] "Generic (PLEG): container finished" podID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerID="cc8c91a07791276d4b7f0e96d7cd1aa010750a16ba1c6db0199ad301d0ba4004" exitCode=0 Sep 10 12:16:38.792851 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.792514 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" event={"ID":"17618a39-7011-4b42-8aeb-5cbeeb925bc9","Type":"ContainerDied","Data":"cc8c91a07791276d4b7f0e96d7cd1aa010750a16ba1c6db0199ad301d0ba4004"} Sep 10 12:16:38.792851 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.792567 2582 scope.go:117] "RemoveContainer" containerID="f9d18451de1a4f4b601210e2c817a0e78b0ee639e303da01d1bafa034001446d" Sep 10 12:16:38.913716 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.913657 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:16:38.972812 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.972784 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-token\") pod \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " Sep 10 12:16:38.972954 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.972821 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-podinfo\") pod \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " Sep 10 12:16:38.972954 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.972864 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-envoy\") pod \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " Sep 10 12:16:38.972954 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.972886 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-credential-socket\") pod \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " Sep 10 12:16:38.972954 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.972913 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-workload-certs\") pod \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " Sep 10 12:16:38.972954 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.972940 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istiod-ca-cert\") pod \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " Sep 10 12:16:38.973245 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.972957 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqmrw\" (UniqueName: \"kubernetes.io/projected/17618a39-7011-4b42-8aeb-5cbeeb925bc9-kube-api-access-zqmrw\") pod \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " Sep 10 12:16:38.973245 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.972993 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-workload-socket\") pod \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " Sep 10 12:16:38.973245 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.973019 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-data\") pod \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\" (UID: \"17618a39-7011-4b42-8aeb-5cbeeb925bc9\") " Sep 10 12:16:38.973382 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.973303 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17618a39-7011-4b42-8aeb-5cbeeb925bc9" (UID: "17618a39-7011-4b42-8aeb-5cbeeb925bc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:16:38.973382 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.973323 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17618a39-7011-4b42-8aeb-5cbeeb925bc9" (UID: "17618a39-7011-4b42-8aeb-5cbeeb925bc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:16:38.973545 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.973509 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-data" (OuterVolumeSpecName: "istio-data") pod "17618a39-7011-4b42-8aeb-5cbeeb925bc9" (UID: "17618a39-7011-4b42-8aeb-5cbeeb925bc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:16:38.973678 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.973611 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17618a39-7011-4b42-8aeb-5cbeeb925bc9" (UID: "17618a39-7011-4b42-8aeb-5cbeeb925bc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:16:38.973745 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.973682 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17618a39-7011-4b42-8aeb-5cbeeb925bc9" (UID: "17618a39-7011-4b42-8aeb-5cbeeb925bc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:16:38.975369 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.975346 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-token" (OuterVolumeSpecName: "istio-token") pod "17618a39-7011-4b42-8aeb-5cbeeb925bc9" (UID: "17618a39-7011-4b42-8aeb-5cbeeb925bc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:16:38.975566 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.975537 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17618a39-7011-4b42-8aeb-5cbeeb925bc9-kube-api-access-zqmrw" (OuterVolumeSpecName: "kube-api-access-zqmrw") pod "17618a39-7011-4b42-8aeb-5cbeeb925bc9" (UID: "17618a39-7011-4b42-8aeb-5cbeeb925bc9"). InnerVolumeSpecName "kube-api-access-zqmrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:16:38.975670 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.975591 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17618a39-7011-4b42-8aeb-5cbeeb925bc9" (UID: "17618a39-7011-4b42-8aeb-5cbeeb925bc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:16:38.975670 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:38.975603 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17618a39-7011-4b42-8aeb-5cbeeb925bc9" (UID: "17618a39-7011-4b42-8aeb-5cbeeb925bc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:16:39.074645 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.074614 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:16:39.074645 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.074641 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqmrw\" (UniqueName: \"kubernetes.io/projected/17618a39-7011-4b42-8aeb-5cbeeb925bc9-kube-api-access-zqmrw\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:16:39.074645 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.074652 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:16:39.074854 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.074661 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:16:39.074854 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.074670 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:16:39.074854 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.074678 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:16:39.074854 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.074686 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:16:39.074854 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.074693 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:16:39.074854 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.074701 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17618a39-7011-4b42-8aeb-5cbeeb925bc9-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:16:39.802231 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.802204 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" Sep 10 12:16:39.802689 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.802194 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc" event={"ID":"17618a39-7011-4b42-8aeb-5cbeeb925bc9","Type":"ContainerDied","Data":"e983ede3e6e59213ad23ce3056e7e028e334752cebcbb684e5f105a80ca0049f"} Sep 10 12:16:39.802689 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.802342 2582 scope.go:117] "RemoveContainer" containerID="cc8c91a07791276d4b7f0e96d7cd1aa010750a16ba1c6db0199ad301d0ba4004" Sep 10 12:16:39.826625 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.826601 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc"] Sep 10 12:16:39.835628 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:39.835603 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-187e7e-openshift-default-5985d788bb-hkfwc"] Sep 10 12:16:40.630752 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:40.630719 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" path="/var/lib/kubelet/pods/17618a39-7011-4b42-8aeb-5cbeeb925bc9/volumes" Sep 10 12:16:45.671029 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:45.670991 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:16:50.026534 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:50.026501 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2"] Sep 10 12:16:50.031176 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:50.031138 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c531cb-nxgq2"] Sep 10 12:16:50.630618 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:50.630573 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="901df1d7-6141-4b81-9512-772565b69d89" path="/var/lib/kubelet/pods/901df1d7-6141-4b81-9512-772565b69d89/volumes" Sep 10 12:16:56.956294 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:56.956220 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg"] Sep 10 12:16:56.956673 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:56.956420 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" containerID="cri-o://1ee0c85ce76fb2c22c2cb2bfe6b3b9ef04898499142746974f65c6caf2e045cc" gracePeriod=30 Sep 10 12:16:57.329554 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.329461 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4"] Sep 10 12:16:57.329992 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.329972 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerName="istio-proxy" Sep 10 12:16:57.330092 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.329994 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerName="istio-proxy" Sep 10 12:16:57.330092 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.330008 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="901df1d7-6141-4b81-9512-772565b69d89" containerName="revoke-keys" Sep 10 12:16:57.330092 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.330016 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="901df1d7-6141-4b81-9512-772565b69d89" containerName="revoke-keys" Sep 10 12:16:57.330092 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.330044 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerName="istio-proxy" Sep 10 12:16:57.330092 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.330052 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerName="istio-proxy" Sep 10 12:16:57.330381 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.330168 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerName="istio-proxy" Sep 10 12:16:57.330381 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.330179 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="901df1d7-6141-4b81-9512-772565b69d89" containerName="revoke-keys" Sep 10 12:16:57.330381 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.330193 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="17618a39-7011-4b42-8aeb-5cbeeb925bc9" containerName="istio-proxy" Sep 10 12:16:57.334586 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.334564 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4" Sep 10 12:16:57.338535 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.338513 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:16:57.338776 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.338763 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:16:57.345679 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.345656 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4"] Sep 10 12:16:57.447310 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.447277 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aeaf3e8-4f75-45d4-ac06-365a6e329a87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4\" (UID: \"2aeaf3e8-4f75-45d4-ac06-365a6e329a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4" Sep 10 12:16:57.548058 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.548022 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aeaf3e8-4f75-45d4-ac06-365a6e329a87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4\" (UID: \"2aeaf3e8-4f75-45d4-ac06-365a6e329a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4" Sep 10 12:16:57.548624 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.548604 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aeaf3e8-4f75-45d4-ac06-365a6e329a87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4\" (UID: \"2aeaf3e8-4f75-45d4-ac06-365a6e329a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4" Sep 10 12:16:57.645642 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.645547 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4" Sep 10 12:16:57.970960 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:57.970935 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4"] Sep 10 12:16:57.972769 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:16:57.972744 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aeaf3e8_4f75_45d4_ac06_365a6e329a87.slice/crio-14d6f69d4fbe240a6f2051ffeca73d2417caa52ed834e991d6d33981605d844f WatchSource:0}: Error finding container 14d6f69d4fbe240a6f2051ffeca73d2417caa52ed834e991d6d33981605d844f: Status 404 returned error can't find the container with id 14d6f69d4fbe240a6f2051ffeca73d2417caa52ed834e991d6d33981605d844f Sep 10 12:16:58.622361 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:58.622329 2582 patch_prober.go:28] interesting pod/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 12:16:58.622502 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:58.622386 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 12:16:58.919570 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:58.919484 2582 generic.go:356] "Generic (PLEG): container finished" podID="2aeaf3e8-4f75-45d4-ac06-365a6e329a87" containerID="ae401c9908bc26f783685f02e7bef2ac87e04688a73e180c3de2b2ce7bebf114" exitCode=0 Sep 10 12:16:58.919707 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:58.919575 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4" event={"ID":"2aeaf3e8-4f75-45d4-ac06-365a6e329a87","Type":"ContainerDied","Data":"ae401c9908bc26f783685f02e7bef2ac87e04688a73e180c3de2b2ce7bebf114"} Sep 10 12:16:58.919707 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:16:58.919626 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4" event={"ID":"2aeaf3e8-4f75-45d4-ac06-365a6e329a87","Type":"ContainerStarted","Data":"14d6f69d4fbe240a6f2051ffeca73d2417caa52ed834e991d6d33981605d844f"} Sep 10 12:17:00.064875 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:00.064845 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4" Sep 10 12:17:00.170617 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:00.170584 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aeaf3e8-4f75-45d4-ac06-365a6e329a87-maas-api-service-ca\") pod \"2aeaf3e8-4f75-45d4-ac06-365a6e329a87\" (UID: \"2aeaf3e8-4f75-45d4-ac06-365a6e329a87\") " Sep 10 12:17:00.170925 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:00.170904 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aeaf3e8-4f75-45d4-ac06-365a6e329a87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2aeaf3e8-4f75-45d4-ac06-365a6e329a87" (UID: "2aeaf3e8-4f75-45d4-ac06-365a6e329a87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:17:00.271847 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:00.271818 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aeaf3e8-4f75-45d4-ac06-365a6e329a87-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:00.931435 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:00.931350 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4" Sep 10 12:17:00.931435 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:00.931349 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4" event={"ID":"2aeaf3e8-4f75-45d4-ac06-365a6e329a87","Type":"ContainerDied","Data":"14d6f69d4fbe240a6f2051ffeca73d2417caa52ed834e991d6d33981605d844f"} Sep 10 12:17:00.931662 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:00.931464 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14d6f69d4fbe240a6f2051ffeca73d2417caa52ed834e991d6d33981605d844f" Sep 10 12:17:01.467732 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:01.467698 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:17:02.221253 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.221190 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:17:02.290565 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.290531 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543d6243-a64a-4935-bc2b-713437ca95b7-istiod-ca-cert\") pod \"543d6243-a64a-4935-bc2b-713437ca95b7\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " Sep 10 12:17:02.290755 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.290577 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-istio-envoy\") pod \"543d6243-a64a-4935-bc2b-713437ca95b7\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " Sep 10 12:17:02.290755 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.290606 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543d6243-a64a-4935-bc2b-713437ca95b7-istio-token\") pod \"543d6243-a64a-4935-bc2b-713437ca95b7\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " Sep 10 12:17:02.290755 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.290628 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-credential-socket\") pod \"543d6243-a64a-4935-bc2b-713437ca95b7\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " Sep 10 12:17:02.290755 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.290652 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543d6243-a64a-4935-bc2b-713437ca95b7-istio-podinfo\") pod \"543d6243-a64a-4935-bc2b-713437ca95b7\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " Sep 10 12:17:02.290755 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.290754 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-workload-socket\") pod \"543d6243-a64a-4935-bc2b-713437ca95b7\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " Sep 10 12:17:02.291046 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.290777 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7kjc\" (UniqueName: \"kubernetes.io/projected/543d6243-a64a-4935-bc2b-713437ca95b7-kube-api-access-r7kjc\") pod \"543d6243-a64a-4935-bc2b-713437ca95b7\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " Sep 10 12:17:02.291046 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.290801 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-istio-data\") pod \"543d6243-a64a-4935-bc2b-713437ca95b7\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " Sep 10 12:17:02.291046 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.290827 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-workload-certs\") pod \"543d6243-a64a-4935-bc2b-713437ca95b7\" (UID: \"543d6243-a64a-4935-bc2b-713437ca95b7\") " Sep 10 12:17:02.291046 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.290936 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543d6243-a64a-4935-bc2b-713437ca95b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "543d6243-a64a-4935-bc2b-713437ca95b7" (UID: "543d6243-a64a-4935-bc2b-713437ca95b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:17:02.291046 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.290985 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "543d6243-a64a-4935-bc2b-713437ca95b7" (UID: "543d6243-a64a-4935-bc2b-713437ca95b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:17:02.291402 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.291185 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "543d6243-a64a-4935-bc2b-713437ca95b7" (UID: "543d6243-a64a-4935-bc2b-713437ca95b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:17:02.291402 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.291282 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:02.291402 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.291299 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543d6243-a64a-4935-bc2b-713437ca95b7-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:02.291402 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.291313 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:02.291622 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.291435 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "543d6243-a64a-4935-bc2b-713437ca95b7" (UID: "543d6243-a64a-4935-bc2b-713437ca95b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:17:02.291622 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.291488 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "543d6243-a64a-4935-bc2b-713437ca95b7" (UID: "543d6243-a64a-4935-bc2b-713437ca95b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:17:02.293185 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.293133 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543d6243-a64a-4935-bc2b-713437ca95b7-kube-api-access-r7kjc" (OuterVolumeSpecName: "kube-api-access-r7kjc") pod "543d6243-a64a-4935-bc2b-713437ca95b7" (UID: "543d6243-a64a-4935-bc2b-713437ca95b7"). InnerVolumeSpecName "kube-api-access-r7kjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:17:02.293338 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.293322 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "543d6243-a64a-4935-bc2b-713437ca95b7" (UID: "543d6243-a64a-4935-bc2b-713437ca95b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:17:02.293486 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.293467 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/543d6243-a64a-4935-bc2b-713437ca95b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "543d6243-a64a-4935-bc2b-713437ca95b7" (UID: "543d6243-a64a-4935-bc2b-713437ca95b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:17:02.293584 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.293486 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543d6243-a64a-4935-bc2b-713437ca95b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "543d6243-a64a-4935-bc2b-713437ca95b7" (UID: "543d6243-a64a-4935-bc2b-713437ca95b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:17:02.392495 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.392456 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:02.392495 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.392494 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7kjc\" (UniqueName: \"kubernetes.io/projected/543d6243-a64a-4935-bc2b-713437ca95b7-kube-api-access-r7kjc\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:02.392689 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.392511 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:02.392689 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.392524 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543d6243-a64a-4935-bc2b-713437ca95b7-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:02.392689 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.392534 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543d6243-a64a-4935-bc2b-713437ca95b7-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:02.392689 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.392545 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543d6243-a64a-4935-bc2b-713437ca95b7-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:02.943906 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.943874 2582 generic.go:356] "Generic (PLEG): container finished" podID="543d6243-a64a-4935-bc2b-713437ca95b7" containerID="1ee0c85ce76fb2c22c2cb2bfe6b3b9ef04898499142746974f65c6caf2e045cc" exitCode=0 Sep 10 12:17:02.944401 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.943926 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" Sep 10 12:17:02.944401 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.943962 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" event={"ID":"543d6243-a64a-4935-bc2b-713437ca95b7","Type":"ContainerDied","Data":"1ee0c85ce76fb2c22c2cb2bfe6b3b9ef04898499142746974f65c6caf2e045cc"} Sep 10 12:17:02.944401 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.944018 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg" event={"ID":"543d6243-a64a-4935-bc2b-713437ca95b7","Type":"ContainerDied","Data":"b06e50244dd4f52acdb9ab5f57fe03b23f53c6d3640f4aa84ef9325014a8c0f9"} Sep 10 12:17:02.944401 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.944040 2582 scope.go:117] "RemoveContainer" containerID="1ee0c85ce76fb2c22c2cb2bfe6b3b9ef04898499142746974f65c6caf2e045cc" Sep 10 12:17:02.953313 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.953297 2582 scope.go:117] "RemoveContainer" containerID="d928df88e5b3d31c67111244a0641c280253e61fb0450e6810f73303ff6fd5d4" Sep 10 12:17:02.961285 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.961267 2582 scope.go:117] "RemoveContainer" containerID="1ee0c85ce76fb2c22c2cb2bfe6b3b9ef04898499142746974f65c6caf2e045cc" Sep 10 12:17:02.961536 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:17:02.961518 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee0c85ce76fb2c22c2cb2bfe6b3b9ef04898499142746974f65c6caf2e045cc\": container with ID starting with 1ee0c85ce76fb2c22c2cb2bfe6b3b9ef04898499142746974f65c6caf2e045cc not found: ID does not exist" containerID="1ee0c85ce76fb2c22c2cb2bfe6b3b9ef04898499142746974f65c6caf2e045cc" Sep 10 12:17:02.961581 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.961544 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee0c85ce76fb2c22c2cb2bfe6b3b9ef04898499142746974f65c6caf2e045cc"} err="failed to get container status \"1ee0c85ce76fb2c22c2cb2bfe6b3b9ef04898499142746974f65c6caf2e045cc\": rpc error: code = NotFound desc = could not find container \"1ee0c85ce76fb2c22c2cb2bfe6b3b9ef04898499142746974f65c6caf2e045cc\": container with ID starting with 1ee0c85ce76fb2c22c2cb2bfe6b3b9ef04898499142746974f65c6caf2e045cc not found: ID does not exist" Sep 10 12:17:02.961581 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.961561 2582 scope.go:117] "RemoveContainer" containerID="d928df88e5b3d31c67111244a0641c280253e61fb0450e6810f73303ff6fd5d4" Sep 10 12:17:02.961830 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:17:02.961815 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d928df88e5b3d31c67111244a0641c280253e61fb0450e6810f73303ff6fd5d4\": container with ID starting with d928df88e5b3d31c67111244a0641c280253e61fb0450e6810f73303ff6fd5d4 not found: ID does not exist" containerID="d928df88e5b3d31c67111244a0641c280253e61fb0450e6810f73303ff6fd5d4" Sep 10 12:17:02.961886 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.961833 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d928df88e5b3d31c67111244a0641c280253e61fb0450e6810f73303ff6fd5d4"} err="failed to get container status \"d928df88e5b3d31c67111244a0641c280253e61fb0450e6810f73303ff6fd5d4\": rpc error: code = NotFound desc = could not find container \"d928df88e5b3d31c67111244a0641c280253e61fb0450e6810f73303ff6fd5d4\": container with ID starting with d928df88e5b3d31c67111244a0641c280253e61fb0450e6810f73303ff6fd5d4 not found: ID does not exist" Sep 10 12:17:02.969406 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.969370 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg"] Sep 10 12:17:02.974922 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:02.974896 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c531cb-openshift-default-f8cd77cd8-vm9lg"] Sep 10 12:17:04.633579 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:04.633548 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" path="/var/lib/kubelet/pods/543d6243-a64a-4935-bc2b-713437ca95b7/volumes" Sep 10 12:17:08.285217 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:08.285181 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4"] Sep 10 12:17:08.291659 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:08.291635 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-639beb-nx5w4"] Sep 10 12:17:08.630286 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:08.630196 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aeaf3e8-4f75-45d4-ac06-365a6e329a87" path="/var/lib/kubelet/pods/2aeaf3e8-4f75-45d4-ac06-365a6e329a87/volumes" Sep 10 12:17:11.160165 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:11.160113 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:17:15.379808 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.379775 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln"] Sep 10 12:17:15.380229 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.380205 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" Sep 10 12:17:15.380229 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.380217 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" Sep 10 12:17:15.380229 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.380228 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" Sep 10 12:17:15.380348 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.380234 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" Sep 10 12:17:15.380348 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.380244 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aeaf3e8-4f75-45d4-ac06-365a6e329a87" containerName="revoke-keys" Sep 10 12:17:15.380348 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.380249 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aeaf3e8-4f75-45d4-ac06-365a6e329a87" containerName="revoke-keys" Sep 10 12:17:15.380348 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.380310 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" Sep 10 12:17:15.380348 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.380319 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aeaf3e8-4f75-45d4-ac06-365a6e329a87" containerName="revoke-keys" Sep 10 12:17:15.384820 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.384801 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln" Sep 10 12:17:15.387625 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.387604 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:17:15.387755 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.387733 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:17:15.392728 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.392393 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln"] Sep 10 12:17:15.524161 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.524106 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7639261-f88f-4ba7-a1fd-80f64b28d2eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln\" (UID: \"e7639261-f88f-4ba7-a1fd-80f64b28d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln" Sep 10 12:17:15.624691 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.624662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7639261-f88f-4ba7-a1fd-80f64b28d2eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln\" (UID: \"e7639261-f88f-4ba7-a1fd-80f64b28d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln" Sep 10 12:17:15.625240 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.625222 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7639261-f88f-4ba7-a1fd-80f64b28d2eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln\" (UID: \"e7639261-f88f-4ba7-a1fd-80f64b28d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln" Sep 10 12:17:15.696100 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.696070 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln" Sep 10 12:17:15.990225 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:15.990050 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln"] Sep 10 12:17:15.992569 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:17:15.992544 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7639261_f88f_4ba7_a1fd_80f64b28d2eb.slice/crio-74f985837d32734e1e1771380eb13b47fa55b62a0dd8a02f55c66acac0c82ed5 WatchSource:0}: Error finding container 74f985837d32734e1e1771380eb13b47fa55b62a0dd8a02f55c66acac0c82ed5: Status 404 returned error can't find the container with id 74f985837d32734e1e1771380eb13b47fa55b62a0dd8a02f55c66acac0c82ed5 Sep 10 12:17:16.048115 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:16.048088 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln" event={"ID":"e7639261-f88f-4ba7-a1fd-80f64b28d2eb","Type":"ContainerStarted","Data":"74f985837d32734e1e1771380eb13b47fa55b62a0dd8a02f55c66acac0c82ed5"} Sep 10 12:17:17.055694 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:17.055599 2582 generic.go:356] "Generic (PLEG): container finished" podID="e7639261-f88f-4ba7-a1fd-80f64b28d2eb" containerID="ec5116ba0c39cb0363a2c079f34c031100d383237a74fd6ec92eec618828958d" exitCode=0 Sep 10 12:17:17.055694 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:17.055683 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln" event={"ID":"e7639261-f88f-4ba7-a1fd-80f64b28d2eb","Type":"ContainerDied","Data":"ec5116ba0c39cb0363a2c079f34c031100d383237a74fd6ec92eec618828958d"} Sep 10 12:17:18.198637 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:18.198613 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln" Sep 10 12:17:18.349673 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:18.349564 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7639261-f88f-4ba7-a1fd-80f64b28d2eb-maas-api-service-ca\") pod \"e7639261-f88f-4ba7-a1fd-80f64b28d2eb\" (UID: \"e7639261-f88f-4ba7-a1fd-80f64b28d2eb\") " Sep 10 12:17:18.349946 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:18.349922 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7639261-f88f-4ba7-a1fd-80f64b28d2eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e7639261-f88f-4ba7-a1fd-80f64b28d2eb" (UID: "e7639261-f88f-4ba7-a1fd-80f64b28d2eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:17:18.451560 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:18.451530 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7639261-f88f-4ba7-a1fd-80f64b28d2eb-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:19.069128 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:19.069102 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln" Sep 10 12:17:19.069315 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:19.069169 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln" event={"ID":"e7639261-f88f-4ba7-a1fd-80f64b28d2eb","Type":"ContainerDied","Data":"74f985837d32734e1e1771380eb13b47fa55b62a0dd8a02f55c66acac0c82ed5"} Sep 10 12:17:19.069315 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:19.069216 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74f985837d32734e1e1771380eb13b47fa55b62a0dd8a02f55c66acac0c82ed5" Sep 10 12:17:26.039548 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:26.039513 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7"] Sep 10 12:17:26.044027 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:26.043998 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c66562-29817375-4vpn7"] Sep 10 12:17:26.305112 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:26.305016 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:17:26.512546 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:26.512511 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln"] Sep 10 12:17:26.527498 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:26.527461 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c66562-dr5ln"] Sep 10 12:17:26.629983 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:26.629904 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1575f8fb-3d05-4ada-9b88-df5698947808" path="/var/lib/kubelet/pods/1575f8fb-3d05-4ada-9b88-df5698947808/volumes" Sep 10 12:17:26.630243 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:26.630227 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7639261-f88f-4ba7-a1fd-80f64b28d2eb" path="/var/lib/kubelet/pods/e7639261-f88f-4ba7-a1fd-80f64b28d2eb/volumes" Sep 10 12:17:32.924757 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:32.924712 2582 scope.go:117] "RemoveContainer" containerID="ffd0cde60c46f8ce1cd4a986ce093012451958dd00104426666d86f800bc31f1" Sep 10 12:17:32.933713 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:32.933693 2582 scope.go:117] "RemoveContainer" containerID="e39d1388a7f27c73dcbe440a9d32d4a4882bff099ec2449ef44973aec09a5370" Sep 10 12:17:33.225735 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:33.225703 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt"] Sep 10 12:17:33.225973 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:33.225948 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" podUID="ac3a5983-e046-4956-aefa-23d9413f861a" containerName="istio-proxy" containerID="cri-o://835059695a73a84dea91fa560eb45199eb428e94b0af1e9d63d395ec5def9f0a" gracePeriod=30 Sep 10 12:17:37.452532 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:37.452498 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:17:38.481842 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.481819 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:17:38.543473 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.543385 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-credential-socket\") pod \"ac3a5983-e046-4956-aefa-23d9413f861a\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " Sep 10 12:17:38.543642 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.543461 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwsv8\" (UniqueName: \"kubernetes.io/projected/ac3a5983-e046-4956-aefa-23d9413f861a-kube-api-access-rwsv8\") pod \"ac3a5983-e046-4956-aefa-23d9413f861a\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " Sep 10 12:17:38.543642 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.543536 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-istio-envoy\") pod \"ac3a5983-e046-4956-aefa-23d9413f861a\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " Sep 10 12:17:38.543642 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.543576 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-istio-data\") pod \"ac3a5983-e046-4956-aefa-23d9413f861a\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " Sep 10 12:17:38.543642 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.543609 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3a5983-e046-4956-aefa-23d9413f861a-istiod-ca-cert\") pod \"ac3a5983-e046-4956-aefa-23d9413f861a\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " Sep 10 12:17:38.543642 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.543639 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac3a5983-e046-4956-aefa-23d9413f861a-istio-token\") pod \"ac3a5983-e046-4956-aefa-23d9413f861a\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " Sep 10 12:17:38.543924 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.543665 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-workload-certs\") pod \"ac3a5983-e046-4956-aefa-23d9413f861a\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " Sep 10 12:17:38.543924 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.543695 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac3a5983-e046-4956-aefa-23d9413f861a" (UID: "ac3a5983-e046-4956-aefa-23d9413f861a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:17:38.543924 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.543714 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-workload-socket\") pod \"ac3a5983-e046-4956-aefa-23d9413f861a\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " Sep 10 12:17:38.543924 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.543778 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3a5983-e046-4956-aefa-23d9413f861a-istio-podinfo\") pod \"ac3a5983-e046-4956-aefa-23d9413f861a\" (UID: \"ac3a5983-e046-4956-aefa-23d9413f861a\") " Sep 10 12:17:38.543924 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.543815 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac3a5983-e046-4956-aefa-23d9413f861a" (UID: "ac3a5983-e046-4956-aefa-23d9413f861a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:17:38.544230 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.543928 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac3a5983-e046-4956-aefa-23d9413f861a" (UID: "ac3a5983-e046-4956-aefa-23d9413f861a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:17:38.544230 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.544027 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac3a5983-e046-4956-aefa-23d9413f861a" (UID: "ac3a5983-e046-4956-aefa-23d9413f861a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:17:38.544230 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.544123 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3a5983-e046-4956-aefa-23d9413f861a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac3a5983-e046-4956-aefa-23d9413f861a" (UID: "ac3a5983-e046-4956-aefa-23d9413f861a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:17:38.544230 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.544174 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:38.544230 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.544193 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:38.544230 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.544205 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:38.544230 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.544217 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:38.545800 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.545777 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac3a5983-e046-4956-aefa-23d9413f861a" (UID: "ac3a5983-e046-4956-aefa-23d9413f861a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:17:38.545954 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.545935 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3a5983-e046-4956-aefa-23d9413f861a-kube-api-access-rwsv8" (OuterVolumeSpecName: "kube-api-access-rwsv8") pod "ac3a5983-e046-4956-aefa-23d9413f861a" (UID: "ac3a5983-e046-4956-aefa-23d9413f861a"). InnerVolumeSpecName "kube-api-access-rwsv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:17:38.546413 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.546397 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac3a5983-e046-4956-aefa-23d9413f861a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac3a5983-e046-4956-aefa-23d9413f861a" (UID: "ac3a5983-e046-4956-aefa-23d9413f861a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:17:38.546485 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.546413 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3a5983-e046-4956-aefa-23d9413f861a-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac3a5983-e046-4956-aefa-23d9413f861a" (UID: "ac3a5983-e046-4956-aefa-23d9413f861a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:17:38.644895 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.644871 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3a5983-e046-4956-aefa-23d9413f861a-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:38.644895 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.644896 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwsv8\" (UniqueName: \"kubernetes.io/projected/ac3a5983-e046-4956-aefa-23d9413f861a-kube-api-access-rwsv8\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:38.645013 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.644905 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3a5983-e046-4956-aefa-23d9413f861a-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:38.645013 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.644914 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3a5983-e046-4956-aefa-23d9413f861a-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:38.645013 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:38.644921 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac3a5983-e046-4956-aefa-23d9413f861a-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:17:39.190006 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:39.189979 2582 generic.go:356] "Generic (PLEG): container finished" podID="ac3a5983-e046-4956-aefa-23d9413f861a" containerID="835059695a73a84dea91fa560eb45199eb428e94b0af1e9d63d395ec5def9f0a" exitCode=0 Sep 10 12:17:39.190302 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:39.190024 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" Sep 10 12:17:39.190302 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:39.190035 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" event={"ID":"ac3a5983-e046-4956-aefa-23d9413f861a","Type":"ContainerDied","Data":"835059695a73a84dea91fa560eb45199eb428e94b0af1e9d63d395ec5def9f0a"} Sep 10 12:17:39.190302 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:39.190076 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt" event={"ID":"ac3a5983-e046-4956-aefa-23d9413f861a","Type":"ContainerDied","Data":"99b39310ceb58b8b2d1eac62e2c014ed04090ff50b7ffaf15b86010ee41b2699"} Sep 10 12:17:39.190302 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:39.190098 2582 scope.go:117] "RemoveContainer" containerID="835059695a73a84dea91fa560eb45199eb428e94b0af1e9d63d395ec5def9f0a" Sep 10 12:17:39.204679 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:39.204654 2582 scope.go:117] "RemoveContainer" containerID="e67568c6f071d99f6699818852be84cdad3871a81fac8fc6231831578c82aecf" Sep 10 12:17:39.210493 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:39.210463 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt"] Sep 10 12:17:39.212445 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:39.212425 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c66562-openshift-default-c69984db8-rp8wt"] Sep 10 12:17:39.213999 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:39.213986 2582 scope.go:117] "RemoveContainer" containerID="835059695a73a84dea91fa560eb45199eb428e94b0af1e9d63d395ec5def9f0a" Sep 10 12:17:39.214295 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:17:39.214271 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835059695a73a84dea91fa560eb45199eb428e94b0af1e9d63d395ec5def9f0a\": container with ID starting with 835059695a73a84dea91fa560eb45199eb428e94b0af1e9d63d395ec5def9f0a not found: ID does not exist" containerID="835059695a73a84dea91fa560eb45199eb428e94b0af1e9d63d395ec5def9f0a" Sep 10 12:17:39.214382 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:39.214301 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835059695a73a84dea91fa560eb45199eb428e94b0af1e9d63d395ec5def9f0a"} err="failed to get container status \"835059695a73a84dea91fa560eb45199eb428e94b0af1e9d63d395ec5def9f0a\": rpc error: code = NotFound desc = could not find container \"835059695a73a84dea91fa560eb45199eb428e94b0af1e9d63d395ec5def9f0a\": container with ID starting with 835059695a73a84dea91fa560eb45199eb428e94b0af1e9d63d395ec5def9f0a not found: ID does not exist" Sep 10 12:17:39.214382 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:39.214319 2582 scope.go:117] "RemoveContainer" containerID="e67568c6f071d99f6699818852be84cdad3871a81fac8fc6231831578c82aecf" Sep 10 12:17:39.214577 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:17:39.214559 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e67568c6f071d99f6699818852be84cdad3871a81fac8fc6231831578c82aecf\": container with ID starting with e67568c6f071d99f6699818852be84cdad3871a81fac8fc6231831578c82aecf not found: ID does not exist" containerID="e67568c6f071d99f6699818852be84cdad3871a81fac8fc6231831578c82aecf" Sep 10 12:17:39.214632 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:39.214587 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e67568c6f071d99f6699818852be84cdad3871a81fac8fc6231831578c82aecf"} err="failed to get container status \"e67568c6f071d99f6699818852be84cdad3871a81fac8fc6231831578c82aecf\": rpc error: code = NotFound desc = could not find container \"e67568c6f071d99f6699818852be84cdad3871a81fac8fc6231831578c82aecf\": container with ID starting with e67568c6f071d99f6699818852be84cdad3871a81fac8fc6231831578c82aecf not found: ID does not exist" Sep 10 12:17:40.636416 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:17:40.636377 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac3a5983-e046-4956-aefa-23d9413f861a" path="/var/lib/kubelet/pods/ac3a5983-e046-4956-aefa-23d9413f861a/volumes" Sep 10 12:18:04.166243 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:04.166208 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:18:17.851844 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:17.851809 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:18:30.184045 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:30.183966 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:18:40.949891 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:40.949858 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:18:55.035493 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.035462 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6"] Sep 10 12:18:55.035955 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.035879 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7639261-f88f-4ba7-a1fd-80f64b28d2eb" containerName="revoke-keys" Sep 10 12:18:55.035955 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.035891 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7639261-f88f-4ba7-a1fd-80f64b28d2eb" containerName="revoke-keys" Sep 10 12:18:55.035955 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.035905 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac3a5983-e046-4956-aefa-23d9413f861a" containerName="istio-proxy" Sep 10 12:18:55.035955 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.035911 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac3a5983-e046-4956-aefa-23d9413f861a" containerName="istio-proxy" Sep 10 12:18:55.035955 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.035918 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac3a5983-e046-4956-aefa-23d9413f861a" containerName="istio-proxy" Sep 10 12:18:55.035955 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.035923 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac3a5983-e046-4956-aefa-23d9413f861a" containerName="istio-proxy" Sep 10 12:18:55.036195 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.036008 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="543d6243-a64a-4935-bc2b-713437ca95b7" containerName="istio-proxy" Sep 10 12:18:55.036195 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.036020 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac3a5983-e046-4956-aefa-23d9413f861a" containerName="istio-proxy" Sep 10 12:18:55.036195 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.036028 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7639261-f88f-4ba7-a1fd-80f64b28d2eb" containerName="revoke-keys" Sep 10 12:18:55.040434 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.040411 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.044824 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.044805 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-742017-openshift-default-dockercfg-4rz4f\"" Sep 10 12:18:55.054844 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.054822 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6"] Sep 10 12:18:55.155942 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.155899 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-token\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.155942 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.155944 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-podinfo\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.156206 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.155982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istiod-ca-cert\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.156206 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.156037 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn6mq\" (UniqueName: \"kubernetes.io/projected/fe6f4c64-aec6-4bc4-bbee-4534b213345e-kube-api-access-tn6mq\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.156206 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.156103 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-credential-socket\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.156206 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.156120 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-workload-socket\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.156206 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.156158 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-data\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.156206 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.156195 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-workload-certs\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.156426 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.156229 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-envoy\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.257545 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.257491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-token\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.257545 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.257546 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-podinfo\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.257801 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.257589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istiod-ca-cert\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.257801 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.257657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn6mq\" (UniqueName: \"kubernetes.io/projected/fe6f4c64-aec6-4bc4-bbee-4534b213345e-kube-api-access-tn6mq\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.257801 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.257731 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-credential-socket\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.257801 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.257756 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-workload-socket\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.257801 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.257787 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-data\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.258059 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.257830 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-workload-certs\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.258059 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.257893 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-envoy\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.258287 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.258259 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-data\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.258287 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.258281 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-credential-socket\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.258454 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.258358 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-workload-certs\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.258532 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.258510 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-workload-socket\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.258635 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.258615 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istiod-ca-cert\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.260212 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.260192 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-podinfo\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.260362 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.260346 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-envoy\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.273488 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.273458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-token\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.275707 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.275686 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn6mq\" (UniqueName: \"kubernetes.io/projected/fe6f4c64-aec6-4bc4-bbee-4534b213345e-kube-api-access-tn6mq\") pod \"e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.350819 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.350735 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:55.633165 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.633123 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6"] Sep 10 12:18:55.635077 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:18:55.635048 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6f4c64_aec6_4bc4_bbee_4534b213345e.slice/crio-3986f401a835ae21ce4f4f1145ed79334e0ce81265bd44e41e5324b1842bc569 WatchSource:0}: Error finding container 3986f401a835ae21ce4f4f1145ed79334e0ce81265bd44e41e5324b1842bc569: Status 404 returned error can't find the container with id 3986f401a835ae21ce4f4f1145ed79334e0ce81265bd44e41e5324b1842bc569 Sep 10 12:18:55.637298 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.637266 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:18:55.637370 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.637331 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:18:55.637370 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:55.637358 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:18:56.621278 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:56.621242 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" event={"ID":"fe6f4c64-aec6-4bc4-bbee-4534b213345e","Type":"ContainerStarted","Data":"f4c89b8ec73a6236bec0033d648cb745ae81dee479f48a08b146b98b3bdb86d0"} Sep 10 12:18:56.621721 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:56.621299 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" event={"ID":"fe6f4c64-aec6-4bc4-bbee-4534b213345e","Type":"ContainerStarted","Data":"3986f401a835ae21ce4f4f1145ed79334e0ce81265bd44e41e5324b1842bc569"} Sep 10 12:18:56.643895 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:56.643848 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" podStartSLOduration=1.6438324149999999 podStartE2EDuration="1.643832415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:18:56.643230435 +0000 UTC m=+1644.522619869" watchObservedRunningTime="2026-09-10 12:18:56.643832415 +0000 UTC m=+1644.523221848" Sep 10 12:18:57.351152 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:57.351107 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:57.356100 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:57.356075 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:57.628440 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:57.628359 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:18:57.629555 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:18:57.629539 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:19:05.666764 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:05.666721 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:19:13.670331 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:13.670287 2582 patch_prober.go:28] interesting pod/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 12:19:13.670851 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:13.670358 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 12:19:16.741545 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:16.741511 2582 generic.go:356] "Generic (PLEG): container finished" podID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerID="f4c89b8ec73a6236bec0033d648cb745ae81dee479f48a08b146b98b3bdb86d0" exitCode=0 Sep 10 12:19:16.741971 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:16.741571 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" event={"ID":"fe6f4c64-aec6-4bc4-bbee-4534b213345e","Type":"ContainerDied","Data":"f4c89b8ec73a6236bec0033d648cb745ae81dee479f48a08b146b98b3bdb86d0"} Sep 10 12:19:16.742019 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:16.741984 2582 scope.go:117] "RemoveContainer" containerID="f4c89b8ec73a6236bec0033d648cb745ae81dee479f48a08b146b98b3bdb86d0" Sep 10 12:19:16.743083 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:16.743055 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:19:16.743156 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:16.743129 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:19:16.743198 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:16.743185 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:19:17.749372 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:17.749338 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" event={"ID":"fe6f4c64-aec6-4bc4-bbee-4534b213345e","Type":"ContainerStarted","Data":"541fb1e3ebb1724b86c8d229b24433cc126c6f4aa51c1aea464d6e606a07a23a"} Sep 10 12:19:18.350919 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:18.350881 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:19:18.351111 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:18.351065 2582 patch_prober.go:28] interesting pod/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 10 12:19:18.351180 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:18.351116 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 10 12:19:19.351899 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:19.351860 2582 patch_prober.go:28] interesting pod/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 10 12:19:19.352334 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:19.351927 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 10 12:19:20.352058 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:20.352024 2582 patch_prober.go:28] interesting pod/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 10 12:19:20.352452 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:20.352090 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 10 12:19:21.351693 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:21.351653 2582 patch_prober.go:28] interesting pod/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 10 12:19:21.351907 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:21.351731 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 10 12:19:22.351697 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:22.351658 2582 patch_prober.go:28] interesting pod/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 10 12:19:22.352091 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:22.351715 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 10 12:19:23.355132 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:23.355101 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:19:23.355538 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:23.355416 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:19:23.356037 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:23.356019 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:19:32.979893 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:32.979862 2582 scope.go:117] "RemoveContainer" containerID="aa557d132ebdb925547bbf1353bfe244bd3ce195ded340e896b5298cd0eeb94e" Sep 10 12:19:40.854405 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:19:40.854372 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:20:03.455377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:03.455283 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:20:10.543892 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.543862 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr"] Sep 10 12:20:10.544586 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.544567 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac3a5983-e046-4956-aefa-23d9413f861a" containerName="istio-proxy" Sep 10 12:20:10.546871 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.546857 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.549678 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.549650 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-2fc5d5-openshift-default-dockercfg-4ls6z\"" Sep 10 12:20:10.563709 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.563688 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr"] Sep 10 12:20:10.581218 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.581196 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-credential-socket\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.581333 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.581253 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-workload-certs\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.581333 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.581311 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxwxj\" (UniqueName: \"kubernetes.io/projected/df333e79-b980-4b24-9cbd-a718472b3e2d-kube-api-access-dxwxj\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.581407 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.581342 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-envoy\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.581407 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.581364 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-token\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.581407 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.581382 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-podinfo\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.581508 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.581464 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-workload-socket\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.581508 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.581493 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-data\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.581573 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.581542 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df333e79-b980-4b24-9cbd-a718472b3e2d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.682654 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.682621 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxwxj\" (UniqueName: \"kubernetes.io/projected/df333e79-b980-4b24-9cbd-a718472b3e2d-kube-api-access-dxwxj\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.682839 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.682666 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-envoy\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.682839 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.682685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-token\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.682839 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.682703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-podinfo\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.682839 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.682746 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-workload-socket\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.682839 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.682762 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-data\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.683111 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.682867 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df333e79-b980-4b24-9cbd-a718472b3e2d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.683111 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.682932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-credential-socket\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.683111 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.683008 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-workload-certs\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.683311 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.683234 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-workload-socket\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.683311 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.683280 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-data\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.683433 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.683370 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-credential-socket\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.683490 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.683453 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-workload-certs\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.683636 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.683618 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df333e79-b980-4b24-9cbd-a718472b3e2d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.685126 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.685103 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-envoy\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.685588 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.685571 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-podinfo\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.692425 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.692402 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-token\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.692497 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.692421 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxwxj\" (UniqueName: \"kubernetes.io/projected/df333e79-b980-4b24-9cbd-a718472b3e2d-kube-api-access-dxwxj\") pod \"e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:10.857077 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:10.856993 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:11.134572 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:11.134547 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr"] Sep 10 12:20:11.136551 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:20:11.136517 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf333e79_b980_4b24_9cbd_a718472b3e2d.slice/crio-a30f58184e6bd5c3935d7c5b762042741d5254baea727295ec675b692da6573a WatchSource:0}: Error finding container a30f58184e6bd5c3935d7c5b762042741d5254baea727295ec675b692da6573a: Status 404 returned error can't find the container with id a30f58184e6bd5c3935d7c5b762042741d5254baea727295ec675b692da6573a Sep 10 12:20:11.138914 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:11.138882 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:20:11.138987 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:11.138946 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:20:11.138987 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:11.138973 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:20:12.060940 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:12.060906 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" event={"ID":"df333e79-b980-4b24-9cbd-a718472b3e2d","Type":"ContainerStarted","Data":"b10261f3b0511d86eef2282d7e35d38aaf60ca9b111ccd939750beb755edf19e"} Sep 10 12:20:12.061355 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:12.060955 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" event={"ID":"df333e79-b980-4b24-9cbd-a718472b3e2d","Type":"ContainerStarted","Data":"a30f58184e6bd5c3935d7c5b762042741d5254baea727295ec675b692da6573a"} Sep 10 12:20:12.083755 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:12.083702 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" podStartSLOduration=2.083686934 podStartE2EDuration="2.083686934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:20:12.080739288 +0000 UTC m=+1719.960128721" watchObservedRunningTime="2026-09-10 12:20:12.083686934 +0000 UTC m=+1719.963076429" Sep 10 12:20:12.857963 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:12.857926 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:12.863547 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:12.863520 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:13.067299 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:13.067264 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:13.068272 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:13.068249 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:23.360788 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:23.360750 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:20:34.194189 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:34.194130 2582 generic.go:356] "Generic (PLEG): container finished" podID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerID="b10261f3b0511d86eef2282d7e35d38aaf60ca9b111ccd939750beb755edf19e" exitCode=0 Sep 10 12:20:34.194604 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:34.194186 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" event={"ID":"df333e79-b980-4b24-9cbd-a718472b3e2d","Type":"ContainerDied","Data":"b10261f3b0511d86eef2282d7e35d38aaf60ca9b111ccd939750beb755edf19e"} Sep 10 12:20:34.194651 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:34.194623 2582 scope.go:117] "RemoveContainer" containerID="b10261f3b0511d86eef2282d7e35d38aaf60ca9b111ccd939750beb755edf19e" Sep 10 12:20:34.195890 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:34.195862 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:20:34.195945 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:34.195933 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:20:34.195981 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:34.195964 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 12:20:35.205958 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:35.205914 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" event={"ID":"df333e79-b980-4b24-9cbd-a718472b3e2d","Type":"ContainerStarted","Data":"5483566a2f65d17e2396450373630127a65e03d280473073243af6aeb576c8a2"} Sep 10 12:20:35.857595 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:35.857559 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:35.857794 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:35.857735 2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 10 12:20:35.857848 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:35.857784 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 10 12:20:36.857693 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:36.857653 2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 10 12:20:36.858072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:36.857715 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 10 12:20:37.858426 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:37.858393 2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 10 12:20:37.858821 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:37.858464 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 10 12:20:38.857658 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:38.857616 2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 10 12:20:38.857838 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:38.857681 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 10 12:20:39.749233 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:39.749187 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f"] Sep 10 12:20:39.751564 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:39.751550 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f" Sep 10 12:20:39.754452 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:39.754430 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:20:39.754588 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:39.754471 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:20:39.763263 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:39.763241 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f"] Sep 10 12:20:39.858030 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:39.857995 2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Sep 10 12:20:39.858219 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:39.858055 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Sep 10 12:20:39.870019 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:39.869989 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ec4969-8907-4634-b14b-97b2e399402d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f\" (UID: \"e9ec4969-8907-4634-b14b-97b2e399402d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f" Sep 10 12:20:39.971479 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:39.971435 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ec4969-8907-4634-b14b-97b2e399402d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f\" (UID: \"e9ec4969-8907-4634-b14b-97b2e399402d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f" Sep 10 12:20:39.972048 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:39.972026 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ec4969-8907-4634-b14b-97b2e399402d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f\" (UID: \"e9ec4969-8907-4634-b14b-97b2e399402d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f" Sep 10 12:20:40.064260 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:40.064172 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f" Sep 10 12:20:40.354470 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:40.354439 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f"] Sep 10 12:20:40.360330 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:20:40.360303 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ec4969_8907_4634_b14b_97b2e399402d.slice/crio-da57ce6528ce12e9e8ce3936fbfafd09b436dc4c8d0553bb6d993cc4e0559709 WatchSource:0}: Error finding container da57ce6528ce12e9e8ce3936fbfafd09b436dc4c8d0553bb6d993cc4e0559709: Status 404 returned error can't find the container with id da57ce6528ce12e9e8ce3936fbfafd09b436dc4c8d0553bb6d993cc4e0559709 Sep 10 12:20:40.888635 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:40.888605 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:40.889024 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:40.889009 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:40.889697 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:40.889678 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:41.257209 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:41.257175 2582 generic.go:356] "Generic (PLEG): container finished" podID="e9ec4969-8907-4634-b14b-97b2e399402d" containerID="fe6e4fd355b8cd6b71be1878f3bf199ed42d10b8a351e187996137b1428def73" exitCode=0 Sep 10 12:20:41.257363 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:41.257261 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f" event={"ID":"e9ec4969-8907-4634-b14b-97b2e399402d","Type":"ContainerDied","Data":"fe6e4fd355b8cd6b71be1878f3bf199ed42d10b8a351e187996137b1428def73"} Sep 10 12:20:41.257363 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:41.257309 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f" event={"ID":"e9ec4969-8907-4634-b14b-97b2e399402d","Type":"ContainerStarted","Data":"da57ce6528ce12e9e8ce3936fbfafd09b436dc4c8d0553bb6d993cc4e0559709"} Sep 10 12:20:42.402027 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:42.402005 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f" Sep 10 12:20:42.496660 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:42.496624 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ec4969-8907-4634-b14b-97b2e399402d-maas-api-service-ca\") pod \"e9ec4969-8907-4634-b14b-97b2e399402d\" (UID: \"e9ec4969-8907-4634-b14b-97b2e399402d\") " Sep 10 12:20:42.496959 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:42.496938 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ec4969-8907-4634-b14b-97b2e399402d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9ec4969-8907-4634-b14b-97b2e399402d" (UID: "e9ec4969-8907-4634-b14b-97b2e399402d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:20:42.597564 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:42.597467 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ec4969-8907-4634-b14b-97b2e399402d-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:20:43.270113 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:43.270086 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f" Sep 10 12:20:43.270320 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:43.270131 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f" event={"ID":"e9ec4969-8907-4634-b14b-97b2e399402d","Type":"ContainerDied","Data":"da57ce6528ce12e9e8ce3936fbfafd09b436dc4c8d0553bb6d993cc4e0559709"} Sep 10 12:20:43.270320 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:43.270193 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da57ce6528ce12e9e8ce3936fbfafd09b436dc4c8d0553bb6d993cc4e0559709" Sep 10 12:20:45.776094 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:45.776055 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f"] Sep 10 12:20:45.779933 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:45.779910 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2fc5d5-62w7f"] Sep 10 12:20:46.630605 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:46.630571 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9ec4969-8907-4634-b14b-97b2e399402d" path="/var/lib/kubelet/pods/e9ec4969-8907-4634-b14b-97b2e399402d/volumes" Sep 10 12:20:49.756160 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:49.756120 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:20:52.382454 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:52.382418 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr"] Sep 10 12:20:52.382865 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:52.382679 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerName="istio-proxy" containerID="cri-o://5483566a2f65d17e2396450373630127a65e03d280473073243af6aeb576c8a2" gracePeriod=30 Sep 10 12:20:55.891257 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:55.891224 2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 12:20:55.891624 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:55.891288 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 12:20:56.166489 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:56.166408 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf"] Sep 10 12:20:56.166819 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:56.166807 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ec4969-8907-4634-b14b-97b2e399402d" containerName="revoke-keys" Sep 10 12:20:56.166864 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:56.166821 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ec4969-8907-4634-b14b-97b2e399402d" containerName="revoke-keys" Sep 10 12:20:56.166901 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:56.166888 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ec4969-8907-4634-b14b-97b2e399402d" containerName="revoke-keys" Sep 10 12:20:56.169006 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:56.168988 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf" Sep 10 12:20:56.171810 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:56.171789 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:20:56.171932 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:56.171789 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:20:56.177119 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:56.176990 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf"] Sep 10 12:20:56.332406 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:56.332371 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1825f568-ad48-4b22-8c57-09febd10a6e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf\" (UID: \"1825f568-ad48-4b22-8c57-09febd10a6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf" Sep 10 12:20:56.433384 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:56.433299 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1825f568-ad48-4b22-8c57-09febd10a6e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf\" (UID: \"1825f568-ad48-4b22-8c57-09febd10a6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf" Sep 10 12:20:56.433943 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:56.433925 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1825f568-ad48-4b22-8c57-09febd10a6e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf\" (UID: \"1825f568-ad48-4b22-8c57-09febd10a6e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf" Sep 10 12:20:56.480378 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:56.480343 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf" Sep 10 12:20:56.752518 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:56.752495 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf"] Sep 10 12:20:56.754355 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:20:56.754327 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1825f568_ad48_4b22_8c57_09febd10a6e7.slice/crio-fc9734562342c18e7853a1ad35eb6a8f4f1a1583a9866457f68e9c1f385d0407 WatchSource:0}: Error finding container fc9734562342c18e7853a1ad35eb6a8f4f1a1583a9866457f68e9c1f385d0407: Status 404 returned error can't find the container with id fc9734562342c18e7853a1ad35eb6a8f4f1a1583a9866457f68e9c1f385d0407 Sep 10 12:20:57.352277 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.352243 2582 generic.go:356] "Generic (PLEG): container finished" podID="1825f568-ad48-4b22-8c57-09febd10a6e7" containerID="a7c518c9dc36ac08c703ae7fbd10b9296ce03b39ecf9d8718a35e98753de0daf" exitCode=0 Sep 10 12:20:57.352681 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.352324 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf" event={"ID":"1825f568-ad48-4b22-8c57-09febd10a6e7","Type":"ContainerDied","Data":"a7c518c9dc36ac08c703ae7fbd10b9296ce03b39ecf9d8718a35e98753de0daf"} Sep 10 12:20:57.352681 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.352377 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf" event={"ID":"1825f568-ad48-4b22-8c57-09febd10a6e7","Type":"ContainerStarted","Data":"fc9734562342c18e7853a1ad35eb6a8f4f1a1583a9866457f68e9c1f385d0407"} Sep 10 12:20:57.636510 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.636489 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:57.644252 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644235 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-workload-certs\") pod \"df333e79-b980-4b24-9cbd-a718472b3e2d\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " Sep 10 12:20:57.644333 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644260 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-envoy\") pod \"df333e79-b980-4b24-9cbd-a718472b3e2d\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " Sep 10 12:20:57.644333 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644282 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-credential-socket\") pod \"df333e79-b980-4b24-9cbd-a718472b3e2d\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " Sep 10 12:20:57.644333 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644315 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-podinfo\") pod \"df333e79-b980-4b24-9cbd-a718472b3e2d\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " Sep 10 12:20:57.644485 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644356 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-token\") pod \"df333e79-b980-4b24-9cbd-a718472b3e2d\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " Sep 10 12:20:57.644485 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644389 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df333e79-b980-4b24-9cbd-a718472b3e2d-istiod-ca-cert\") pod \"df333e79-b980-4b24-9cbd-a718472b3e2d\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " Sep 10 12:20:57.644485 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644416 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-data\") pod \"df333e79-b980-4b24-9cbd-a718472b3e2d\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " Sep 10 12:20:57.644485 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644441 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-workload-socket\") pod \"df333e79-b980-4b24-9cbd-a718472b3e2d\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " Sep 10 12:20:57.644485 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644470 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxwxj\" (UniqueName: \"kubernetes.io/projected/df333e79-b980-4b24-9cbd-a718472b3e2d-kube-api-access-dxwxj\") pod \"df333e79-b980-4b24-9cbd-a718472b3e2d\" (UID: \"df333e79-b980-4b24-9cbd-a718472b3e2d\") " Sep 10 12:20:57.644751 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644491 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df333e79-b980-4b24-9cbd-a718472b3e2d" (UID: "df333e79-b980-4b24-9cbd-a718472b3e2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:20:57.644751 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644618 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df333e79-b980-4b24-9cbd-a718472b3e2d" (UID: "df333e79-b980-4b24-9cbd-a718472b3e2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:20:57.644751 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644719 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-data" (OuterVolumeSpecName: "istio-data") pod "df333e79-b980-4b24-9cbd-a718472b3e2d" (UID: "df333e79-b980-4b24-9cbd-a718472b3e2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:20:57.644924 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644833 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:20:57.644924 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644852 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:20:57.644924 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644865 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:20:57.644924 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644828 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df333e79-b980-4b24-9cbd-a718472b3e2d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df333e79-b980-4b24-9cbd-a718472b3e2d" (UID: "df333e79-b980-4b24-9cbd-a718472b3e2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:20:57.644924 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.644920 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df333e79-b980-4b24-9cbd-a718472b3e2d" (UID: "df333e79-b980-4b24-9cbd-a718472b3e2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:20:57.646628 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.646601 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df333e79-b980-4b24-9cbd-a718472b3e2d" (UID: "df333e79-b980-4b24-9cbd-a718472b3e2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:20:57.646628 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.646609 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-token" (OuterVolumeSpecName: "istio-token") pod "df333e79-b980-4b24-9cbd-a718472b3e2d" (UID: "df333e79-b980-4b24-9cbd-a718472b3e2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:20:57.646761 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.646644 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df333e79-b980-4b24-9cbd-a718472b3e2d" (UID: "df333e79-b980-4b24-9cbd-a718472b3e2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:20:57.646856 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.646841 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df333e79-b980-4b24-9cbd-a718472b3e2d-kube-api-access-dxwxj" (OuterVolumeSpecName: "kube-api-access-dxwxj") pod "df333e79-b980-4b24-9cbd-a718472b3e2d" (UID: "df333e79-b980-4b24-9cbd-a718472b3e2d"). InnerVolumeSpecName "kube-api-access-dxwxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:20:57.745430 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.745398 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:20:57.745430 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.745425 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:20:57.745430 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.745433 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df333e79-b980-4b24-9cbd-a718472b3e2d-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:20:57.745665 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.745441 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df333e79-b980-4b24-9cbd-a718472b3e2d-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:20:57.745665 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.745450 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df333e79-b980-4b24-9cbd-a718472b3e2d-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:20:57.745665 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:57.745458 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxwxj\" (UniqueName: \"kubernetes.io/projected/df333e79-b980-4b24-9cbd-a718472b3e2d-kube-api-access-dxwxj\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:20:58.360559 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.360528 2582 generic.go:356] "Generic (PLEG): container finished" podID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerID="5483566a2f65d17e2396450373630127a65e03d280473073243af6aeb576c8a2" exitCode=0 Sep 10 12:20:58.361009 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.360580 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" Sep 10 12:20:58.361009 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.360613 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" event={"ID":"df333e79-b980-4b24-9cbd-a718472b3e2d","Type":"ContainerDied","Data":"5483566a2f65d17e2396450373630127a65e03d280473073243af6aeb576c8a2"} Sep 10 12:20:58.361009 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.360664 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr" event={"ID":"df333e79-b980-4b24-9cbd-a718472b3e2d","Type":"ContainerDied","Data":"a30f58184e6bd5c3935d7c5b762042741d5254baea727295ec675b692da6573a"} Sep 10 12:20:58.361009 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.360685 2582 scope.go:117] "RemoveContainer" containerID="5483566a2f65d17e2396450373630127a65e03d280473073243af6aeb576c8a2" Sep 10 12:20:58.372600 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.372584 2582 scope.go:117] "RemoveContainer" containerID="b10261f3b0511d86eef2282d7e35d38aaf60ca9b111ccd939750beb755edf19e" Sep 10 12:20:58.381942 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.381918 2582 scope.go:117] "RemoveContainer" containerID="5483566a2f65d17e2396450373630127a65e03d280473073243af6aeb576c8a2" Sep 10 12:20:58.382349 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:20:58.382323 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5483566a2f65d17e2396450373630127a65e03d280473073243af6aeb576c8a2\": container with ID starting with 5483566a2f65d17e2396450373630127a65e03d280473073243af6aeb576c8a2 not found: ID does not exist" containerID="5483566a2f65d17e2396450373630127a65e03d280473073243af6aeb576c8a2" Sep 10 12:20:58.382425 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.382361 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5483566a2f65d17e2396450373630127a65e03d280473073243af6aeb576c8a2"} err="failed to get container status \"5483566a2f65d17e2396450373630127a65e03d280473073243af6aeb576c8a2\": rpc error: code = NotFound desc = could not find container \"5483566a2f65d17e2396450373630127a65e03d280473073243af6aeb576c8a2\": container with ID starting with 5483566a2f65d17e2396450373630127a65e03d280473073243af6aeb576c8a2 not found: ID does not exist" Sep 10 12:20:58.382425 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.382384 2582 scope.go:117] "RemoveContainer" containerID="b10261f3b0511d86eef2282d7e35d38aaf60ca9b111ccd939750beb755edf19e" Sep 10 12:20:58.382701 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:20:58.382681 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10261f3b0511d86eef2282d7e35d38aaf60ca9b111ccd939750beb755edf19e\": container with ID starting with b10261f3b0511d86eef2282d7e35d38aaf60ca9b111ccd939750beb755edf19e not found: ID does not exist" containerID="b10261f3b0511d86eef2282d7e35d38aaf60ca9b111ccd939750beb755edf19e" Sep 10 12:20:58.382762 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.382710 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10261f3b0511d86eef2282d7e35d38aaf60ca9b111ccd939750beb755edf19e"} err="failed to get container status \"b10261f3b0511d86eef2282d7e35d38aaf60ca9b111ccd939750beb755edf19e\": rpc error: code = NotFound desc = could not find container \"b10261f3b0511d86eef2282d7e35d38aaf60ca9b111ccd939750beb755edf19e\": container with ID starting with b10261f3b0511d86eef2282d7e35d38aaf60ca9b111ccd939750beb755edf19e not found: ID does not exist" Sep 10 12:20:58.384216 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.384198 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr"] Sep 10 12:20:58.387401 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.387381 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2fc5d5-openshift-default-84677f7d58-9rmwr"] Sep 10 12:20:58.488570 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.488543 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf" Sep 10 12:20:58.552302 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.552272 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1825f568-ad48-4b22-8c57-09febd10a6e7-maas-api-service-ca\") pod \"1825f568-ad48-4b22-8c57-09febd10a6e7\" (UID: \"1825f568-ad48-4b22-8c57-09febd10a6e7\") " Sep 10 12:20:58.552582 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.552560 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1825f568-ad48-4b22-8c57-09febd10a6e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1825f568-ad48-4b22-8c57-09febd10a6e7" (UID: "1825f568-ad48-4b22-8c57-09febd10a6e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:20:58.631119 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.631041 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" path="/var/lib/kubelet/pods/df333e79-b980-4b24-9cbd-a718472b3e2d/volumes" Sep 10 12:20:58.653429 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:58.653404 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1825f568-ad48-4b22-8c57-09febd10a6e7-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:20:59.368273 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:59.368246 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf" Sep 10 12:20:59.368691 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:59.368294 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf" event={"ID":"1825f568-ad48-4b22-8c57-09febd10a6e7","Type":"ContainerDied","Data":"fc9734562342c18e7853a1ad35eb6a8f4f1a1583a9866457f68e9c1f385d0407"} Sep 10 12:20:59.368691 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:20:59.368348 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc9734562342c18e7853a1ad35eb6a8f4f1a1583a9866457f68e9c1f385d0407" Sep 10 12:21:09.263000 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:21:09.262964 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:21:11.831820 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:21:11.831785 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf"] Sep 10 12:21:11.834730 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:21:11.834703 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19f6c3-7rfqf"] Sep 10 12:21:12.630186 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:21:12.630137 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1825f568-ad48-4b22-8c57-09febd10a6e7" path="/var/lib/kubelet/pods/1825f568-ad48-4b22-8c57-09febd10a6e7/volumes" Sep 10 12:21:22.164816 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:21:22.164780 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:21:32.740011 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:21:32.739985 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 12:21:32.744188 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:21:32.744167 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 12:21:33.012387 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:21:33.012299 2582 scope.go:117] "RemoveContainer" containerID="7a3ff55a3a7800ec0af812f146ad6718afb1e7ba31cbbd73aa7babcf3e5d3bba" Sep 10 12:22:00.720647 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.720620 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9"] Sep 10 12:22:00.721109 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.721051 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerName="istio-proxy" Sep 10 12:22:00.721109 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.721063 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerName="istio-proxy" Sep 10 12:22:00.721109 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.721075 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1825f568-ad48-4b22-8c57-09febd10a6e7" containerName="revoke-keys" Sep 10 12:22:00.721109 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.721081 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1825f568-ad48-4b22-8c57-09febd10a6e7" containerName="revoke-keys" Sep 10 12:22:00.721271 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.721183 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerName="istio-proxy" Sep 10 12:22:00.721271 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.721196 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerName="istio-proxy" Sep 10 12:22:00.721271 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.721205 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1825f568-ad48-4b22-8c57-09febd10a6e7" containerName="revoke-keys" Sep 10 12:22:00.724371 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.724353 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9" Sep 10 12:22:00.727619 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.727598 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:22:00.727761 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.727746 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:22:00.738610 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.738585 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9"] Sep 10 12:22:00.843921 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.843884 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b77cc9-f1ee-4ac3-b3b2-c722de06a452-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9\" (UID: \"c2b77cc9-f1ee-4ac3-b3b2-c722de06a452\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9" Sep 10 12:22:00.945015 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.944980 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b77cc9-f1ee-4ac3-b3b2-c722de06a452-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9\" (UID: \"c2b77cc9-f1ee-4ac3-b3b2-c722de06a452\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9" Sep 10 12:22:00.945697 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:00.945676 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b77cc9-f1ee-4ac3-b3b2-c722de06a452-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9\" (UID: \"c2b77cc9-f1ee-4ac3-b3b2-c722de06a452\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9" Sep 10 12:22:01.034006 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:01.033915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9" Sep 10 12:22:01.316741 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:01.316716 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9"] Sep 10 12:22:01.318880 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:22:01.318850 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b77cc9_f1ee_4ac3_b3b2_c722de06a452.slice/crio-b6599cec6237d2149e4d08abd96cfab9942214d0f4750b8e2baef99d469ceba6 WatchSource:0}: Error finding container b6599cec6237d2149e4d08abd96cfab9942214d0f4750b8e2baef99d469ceba6: Status 404 returned error can't find the container with id b6599cec6237d2149e4d08abd96cfab9942214d0f4750b8e2baef99d469ceba6 Sep 10 12:22:01.729467 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:01.729431 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9" event={"ID":"c2b77cc9-f1ee-4ac3-b3b2-c722de06a452","Type":"ContainerStarted","Data":"e789ac4bdeb3a48c1fc4458a7486dac959de5707e35f7d602c82856857467bd9"} Sep 10 12:22:01.729887 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:01.729521 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9" event={"ID":"c2b77cc9-f1ee-4ac3-b3b2-c722de06a452","Type":"ContainerStarted","Data":"b6599cec6237d2149e4d08abd96cfab9942214d0f4750b8e2baef99d469ceba6"} Sep 10 12:22:01.747641 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:01.747589 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9" podStartSLOduration=1.747575162 podStartE2EDuration="1.747575162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:22:01.74585898 +0000 UTC m=+1829.625248412" watchObservedRunningTime="2026-09-10 12:22:01.747575162 +0000 UTC m=+1829.626964593" Sep 10 12:22:02.737098 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:02.737063 2582 generic.go:356] "Generic (PLEG): container finished" podID="c2b77cc9-f1ee-4ac3-b3b2-c722de06a452" containerID="e789ac4bdeb3a48c1fc4458a7486dac959de5707e35f7d602c82856857467bd9" exitCode=0 Sep 10 12:22:02.737526 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:02.737165 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9" event={"ID":"c2b77cc9-f1ee-4ac3-b3b2-c722de06a452","Type":"ContainerDied","Data":"e789ac4bdeb3a48c1fc4458a7486dac959de5707e35f7d602c82856857467bd9"} Sep 10 12:22:03.885448 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:03.885420 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9" Sep 10 12:22:03.977785 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:03.977752 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b77cc9-f1ee-4ac3-b3b2-c722de06a452-maas-api-service-ca\") pod \"c2b77cc9-f1ee-4ac3-b3b2-c722de06a452\" (UID: \"c2b77cc9-f1ee-4ac3-b3b2-c722de06a452\") " Sep 10 12:22:03.978100 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:03.978078 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b77cc9-f1ee-4ac3-b3b2-c722de06a452-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2b77cc9-f1ee-4ac3-b3b2-c722de06a452" (UID: "c2b77cc9-f1ee-4ac3-b3b2-c722de06a452"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:22:04.079339 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:04.079258 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b77cc9-f1ee-4ac3-b3b2-c722de06a452-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:22:04.750255 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:04.750230 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9" Sep 10 12:22:04.750255 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:04.750242 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9" event={"ID":"c2b77cc9-f1ee-4ac3-b3b2-c722de06a452","Type":"ContainerDied","Data":"b6599cec6237d2149e4d08abd96cfab9942214d0f4750b8e2baef99d469ceba6"} Sep 10 12:22:04.750455 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:04.750290 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6599cec6237d2149e4d08abd96cfab9942214d0f4750b8e2baef99d469ceba6" Sep 10 12:22:11.678205 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:11.678175 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9"] Sep 10 12:22:11.687669 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:11.687641 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-742017-4b9b9"] Sep 10 12:22:12.629956 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:12.629924 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2b77cc9-f1ee-4ac3-b3b2-c722de06a452" path="/var/lib/kubelet/pods/c2b77cc9-f1ee-4ac3-b3b2-c722de06a452/volumes" Sep 10 12:22:14.585243 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:14.585212 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:22:19.430927 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.430891 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6"] Sep 10 12:22:19.431368 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.431118 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" containerID="cri-o://541fb1e3ebb1724b86c8d229b24433cc126c6f4aa51c1aea464d6e606a07a23a" gracePeriod=30 Sep 10 12:22:19.771247 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.771211 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj"] Sep 10 12:22:19.771649 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.771636 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b77cc9-f1ee-4ac3-b3b2-c722de06a452" containerName="revoke-keys" Sep 10 12:22:19.771697 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.771652 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b77cc9-f1ee-4ac3-b3b2-c722de06a452" containerName="revoke-keys" Sep 10 12:22:19.771697 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.771667 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerName="istio-proxy" Sep 10 12:22:19.771697 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.771676 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="df333e79-b980-4b24-9cbd-a718472b3e2d" containerName="istio-proxy" Sep 10 12:22:19.771791 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.771748 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2b77cc9-f1ee-4ac3-b3b2-c722de06a452" containerName="revoke-keys" Sep 10 12:22:19.775180 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.775157 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj" Sep 10 12:22:19.778037 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.778009 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:22:19.778378 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.778357 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:22:19.786726 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.786701 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj"] Sep 10 12:22:19.823337 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.823307 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f64a8e-ebf8-44e5-8745-64f52c11f58e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj\" (UID: \"57f64a8e-ebf8-44e5-8745-64f52c11f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj" Sep 10 12:22:19.923795 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.923749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f64a8e-ebf8-44e5-8745-64f52c11f58e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj\" (UID: \"57f64a8e-ebf8-44e5-8745-64f52c11f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj" Sep 10 12:22:19.924502 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:19.924480 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f64a8e-ebf8-44e5-8745-64f52c11f58e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj\" (UID: \"57f64a8e-ebf8-44e5-8745-64f52c11f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj" Sep 10 12:22:20.087003 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:20.086911 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj" Sep 10 12:22:20.370437 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:20.370407 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj"] Sep 10 12:22:20.371652 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:22:20.371627 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f64a8e_ebf8_44e5_8745_64f52c11f58e.slice/crio-5812db0ba0910a8ce1dd5a08d8a091a093966abcb3e01bb0b376aad2ec6a3b41 WatchSource:0}: Error finding container 5812db0ba0910a8ce1dd5a08d8a091a093966abcb3e01bb0b376aad2ec6a3b41: Status 404 returned error can't find the container with id 5812db0ba0910a8ce1dd5a08d8a091a093966abcb3e01bb0b376aad2ec6a3b41 Sep 10 12:22:20.848488 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:20.848449 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj" event={"ID":"57f64a8e-ebf8-44e5-8745-64f52c11f58e","Type":"ContainerStarted","Data":"b010a7a22ee8444a240466ae0c917c75009bee0e4639da1adedfd3902966918f"} Sep 10 12:22:20.848870 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:20.848510 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj" event={"ID":"57f64a8e-ebf8-44e5-8745-64f52c11f58e","Type":"ContainerStarted","Data":"5812db0ba0910a8ce1dd5a08d8a091a093966abcb3e01bb0b376aad2ec6a3b41"} Sep 10 12:22:20.872182 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:20.872114 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj" podStartSLOduration=1.872100336 podStartE2EDuration="1.872100336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:22:20.870217276 +0000 UTC m=+1848.749606708" watchObservedRunningTime="2026-09-10 12:22:20.872100336 +0000 UTC m=+1848.751489805" Sep 10 12:22:21.856970 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:21.856934 2582 generic.go:356] "Generic (PLEG): container finished" podID="57f64a8e-ebf8-44e5-8745-64f52c11f58e" containerID="b010a7a22ee8444a240466ae0c917c75009bee0e4639da1adedfd3902966918f" exitCode=0 Sep 10 12:22:21.857380 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:21.857004 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj" event={"ID":"57f64a8e-ebf8-44e5-8745-64f52c11f58e","Type":"ContainerDied","Data":"b010a7a22ee8444a240466ae0c917c75009bee0e4639da1adedfd3902966918f"} Sep 10 12:22:23.007643 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:23.007607 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj" Sep 10 12:22:23.053283 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:23.053252 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f64a8e-ebf8-44e5-8745-64f52c11f58e-maas-api-service-ca\") pod \"57f64a8e-ebf8-44e5-8745-64f52c11f58e\" (UID: \"57f64a8e-ebf8-44e5-8745-64f52c11f58e\") " Sep 10 12:22:23.053599 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:23.053577 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f64a8e-ebf8-44e5-8745-64f52c11f58e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "57f64a8e-ebf8-44e5-8745-64f52c11f58e" (UID: "57f64a8e-ebf8-44e5-8745-64f52c11f58e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:22:23.154635 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:23.154554 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f64a8e-ebf8-44e5-8745-64f52c11f58e-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:22:23.358036 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:23.357995 2582 patch_prober.go:28] interesting pod/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 12:22:23.358232 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:23.358064 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 12:22:23.871836 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:23.871811 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj" Sep 10 12:22:23.872048 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:23.871804 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj" event={"ID":"57f64a8e-ebf8-44e5-8745-64f52c11f58e","Type":"ContainerDied","Data":"5812db0ba0910a8ce1dd5a08d8a091a093966abcb3e01bb0b376aad2ec6a3b41"} Sep 10 12:22:23.872048 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:23.871928 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5812db0ba0910a8ce1dd5a08d8a091a093966abcb3e01bb0b376aad2ec6a3b41" Sep 10 12:22:24.688008 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.687986 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:22:24.769587 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.769509 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istiod-ca-cert\") pod \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " Sep 10 12:22:24.769587 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.769542 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-credential-socket\") pod \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " Sep 10 12:22:24.769587 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.769559 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-workload-socket\") pod \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " Sep 10 12:22:24.769853 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.769590 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-podinfo\") pod \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " Sep 10 12:22:24.769853 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.769623 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-workload-certs\") pod \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " Sep 10 12:22:24.769853 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.769675 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-envoy\") pod \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " Sep 10 12:22:24.769853 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.769722 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn6mq\" (UniqueName: \"kubernetes.io/projected/fe6f4c64-aec6-4bc4-bbee-4534b213345e-kube-api-access-tn6mq\") pod \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " Sep 10 12:22:24.769853 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.769783 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-token\") pod \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " Sep 10 12:22:24.769853 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.769817 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-data\") pod \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\" (UID: \"fe6f4c64-aec6-4bc4-bbee-4534b213345e\") " Sep 10 12:22:24.769853 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.769839 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe6f4c64-aec6-4bc4-bbee-4534b213345e" (UID: "fe6f4c64-aec6-4bc4-bbee-4534b213345e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:22:24.770102 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.769861 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe6f4c64-aec6-4bc4-bbee-4534b213345e" (UID: "fe6f4c64-aec6-4bc4-bbee-4534b213345e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:22:24.770102 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.769947 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe6f4c64-aec6-4bc4-bbee-4534b213345e" (UID: "fe6f4c64-aec6-4bc4-bbee-4534b213345e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:22:24.770102 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.769979 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe6f4c64-aec6-4bc4-bbee-4534b213345e" (UID: "fe6f4c64-aec6-4bc4-bbee-4534b213345e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:22:24.770286 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.770194 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe6f4c64-aec6-4bc4-bbee-4534b213345e" (UID: "fe6f4c64-aec6-4bc4-bbee-4534b213345e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:22:24.770286 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.770273 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:22:24.770389 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.770302 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:22:24.770389 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.770319 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:22:24.770389 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.770332 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:22:24.772177 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.772122 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe6f4c64-aec6-4bc4-bbee-4534b213345e" (UID: "fe6f4c64-aec6-4bc4-bbee-4534b213345e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:22:24.772177 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.772162 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe6f4c64-aec6-4bc4-bbee-4534b213345e" (UID: "fe6f4c64-aec6-4bc4-bbee-4534b213345e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:22:24.772346 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.772176 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6f4c64-aec6-4bc4-bbee-4534b213345e-kube-api-access-tn6mq" (OuterVolumeSpecName: "kube-api-access-tn6mq") pod "fe6f4c64-aec6-4bc4-bbee-4534b213345e" (UID: "fe6f4c64-aec6-4bc4-bbee-4534b213345e"). InnerVolumeSpecName "kube-api-access-tn6mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:22:24.772346 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.772304 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe6f4c64-aec6-4bc4-bbee-4534b213345e" (UID: "fe6f4c64-aec6-4bc4-bbee-4534b213345e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:22:24.870971 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.870932 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:22:24.870971 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.870964 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:22:24.870971 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.870976 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:22:24.871239 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.870986 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe6f4c64-aec6-4bc4-bbee-4534b213345e-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:22:24.871239 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.871000 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn6mq\" (UniqueName: \"kubernetes.io/projected/fe6f4c64-aec6-4bc4-bbee-4534b213345e-kube-api-access-tn6mq\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:22:24.879514 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.879486 2582 generic.go:356] "Generic (PLEG): container finished" podID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerID="541fb1e3ebb1724b86c8d229b24433cc126c6f4aa51c1aea464d6e606a07a23a" exitCode=0 Sep 10 12:22:24.879630 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.879534 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" Sep 10 12:22:24.879630 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.879567 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" event={"ID":"fe6f4c64-aec6-4bc4-bbee-4534b213345e","Type":"ContainerDied","Data":"541fb1e3ebb1724b86c8d229b24433cc126c6f4aa51c1aea464d6e606a07a23a"} Sep 10 12:22:24.879714 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.879631 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6" event={"ID":"fe6f4c64-aec6-4bc4-bbee-4534b213345e","Type":"ContainerDied","Data":"3986f401a835ae21ce4f4f1145ed79334e0ce81265bd44e41e5324b1842bc569"} Sep 10 12:22:24.879714 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.879656 2582 scope.go:117] "RemoveContainer" containerID="541fb1e3ebb1724b86c8d229b24433cc126c6f4aa51c1aea464d6e606a07a23a" Sep 10 12:22:24.889591 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.889545 2582 scope.go:117] "RemoveContainer" containerID="f4c89b8ec73a6236bec0033d648cb745ae81dee479f48a08b146b98b3bdb86d0" Sep 10 12:22:24.898420 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.898403 2582 scope.go:117] "RemoveContainer" containerID="541fb1e3ebb1724b86c8d229b24433cc126c6f4aa51c1aea464d6e606a07a23a" Sep 10 12:22:24.898716 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:22:24.898683 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541fb1e3ebb1724b86c8d229b24433cc126c6f4aa51c1aea464d6e606a07a23a\": container with ID starting with 541fb1e3ebb1724b86c8d229b24433cc126c6f4aa51c1aea464d6e606a07a23a not found: ID does not exist" containerID="541fb1e3ebb1724b86c8d229b24433cc126c6f4aa51c1aea464d6e606a07a23a" Sep 10 12:22:24.898769 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.898725 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541fb1e3ebb1724b86c8d229b24433cc126c6f4aa51c1aea464d6e606a07a23a"} err="failed to get container status \"541fb1e3ebb1724b86c8d229b24433cc126c6f4aa51c1aea464d6e606a07a23a\": rpc error: code = NotFound desc = could not find container \"541fb1e3ebb1724b86c8d229b24433cc126c6f4aa51c1aea464d6e606a07a23a\": container with ID starting with 541fb1e3ebb1724b86c8d229b24433cc126c6f4aa51c1aea464d6e606a07a23a not found: ID does not exist" Sep 10 12:22:24.898769 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.898745 2582 scope.go:117] "RemoveContainer" containerID="f4c89b8ec73a6236bec0033d648cb745ae81dee479f48a08b146b98b3bdb86d0" Sep 10 12:22:24.898996 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:22:24.898972 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c89b8ec73a6236bec0033d648cb745ae81dee479f48a08b146b98b3bdb86d0\": container with ID starting with f4c89b8ec73a6236bec0033d648cb745ae81dee479f48a08b146b98b3bdb86d0 not found: ID does not exist" containerID="f4c89b8ec73a6236bec0033d648cb745ae81dee479f48a08b146b98b3bdb86d0" Sep 10 12:22:24.899040 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.899002 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c89b8ec73a6236bec0033d648cb745ae81dee479f48a08b146b98b3bdb86d0"} err="failed to get container status \"f4c89b8ec73a6236bec0033d648cb745ae81dee479f48a08b146b98b3bdb86d0\": rpc error: code = NotFound desc = could not find container \"f4c89b8ec73a6236bec0033d648cb745ae81dee479f48a08b146b98b3bdb86d0\": container with ID starting with f4c89b8ec73a6236bec0033d648cb745ae81dee479f48a08b146b98b3bdb86d0 not found: ID does not exist" Sep 10 12:22:24.912965 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.912942 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6"] Sep 10 12:22:24.923204 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:24.923182 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-742017-openshift-default-6c5b96c546-s8kk6"] Sep 10 12:22:26.630668 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:26.630640 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" path="/var/lib/kubelet/pods/fe6f4c64-aec6-4bc4-bbee-4534b213345e/volumes" Sep 10 12:22:30.585792 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:30.585757 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn"] Sep 10 12:22:30.595172 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:30.595107 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c07be0-29817375-fzlmn"] Sep 10 12:22:30.633384 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:30.633352 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="163819d9-5ad9-4076-9b79-453a90857a39" path="/var/lib/kubelet/pods/163819d9-5ad9-4076-9b79-453a90857a39/volumes" Sep 10 12:22:30.911094 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:30.911007 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj"] Sep 10 12:22:30.914222 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:30.914198 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c07be0-vghrj"] Sep 10 12:22:32.631725 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:32.631689 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57f64a8e-ebf8-44e5-8745-64f52c11f58e" path="/var/lib/kubelet/pods/57f64a8e-ebf8-44e5-8745-64f52c11f58e/volumes" Sep 10 12:22:33.037708 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:33.037680 2582 scope.go:117] "RemoveContainer" containerID="01cc07bf53e7d8ba337162accb1a0236096f1e7041fc7a9a2dafc586008124d3" Sep 10 12:22:33.046749 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:33.046728 2582 scope.go:117] "RemoveContainer" containerID="e682c4f0fd716db7834a1a0d9b07870e1ee97752f64e6cf4871f7a1da6b83c78" Sep 10 12:22:33.055218 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:33.055196 2582 scope.go:117] "RemoveContainer" containerID="d378c9dda53a652b686664fa04ede5f2d81db6b99aa8862a456c8b2f2f063e04" Sep 10 12:22:33.861867 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:33.861817 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:22:38.890432 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:38.890395 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc"] Sep 10 12:22:38.890808 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:38.890772 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f64a8e-ebf8-44e5-8745-64f52c11f58e" containerName="revoke-keys" Sep 10 12:22:38.890808 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:38.890782 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f64a8e-ebf8-44e5-8745-64f52c11f58e" containerName="revoke-keys" Sep 10 12:22:38.890808 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:38.890794 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" Sep 10 12:22:38.890808 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:38.890799 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" Sep 10 12:22:38.890943 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:38.890873 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" Sep 10 12:22:38.890943 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:38.890882 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" Sep 10 12:22:38.890943 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:38.890892 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="57f64a8e-ebf8-44e5-8745-64f52c11f58e" containerName="revoke-keys" Sep 10 12:22:38.894106 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:38.894090 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc" Sep 10 12:22:38.897385 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:38.897361 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 12:22:38.897572 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:38.897542 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\"" Sep 10 12:22:38.908539 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:38.908509 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc"] Sep 10 12:22:38.995167 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:38.995101 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9030356a-4c10-4a92-a97d-2d884c6ef400-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc\" (UID: \"9030356a-4c10-4a92-a97d-2d884c6ef400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc" Sep 10 12:22:39.096281 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:39.096248 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9030356a-4c10-4a92-a97d-2d884c6ef400-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc\" (UID: \"9030356a-4c10-4a92-a97d-2d884c6ef400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc" Sep 10 12:22:39.096866 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:39.096847 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9030356a-4c10-4a92-a97d-2d884c6ef400-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc\" (UID: \"9030356a-4c10-4a92-a97d-2d884c6ef400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc" Sep 10 12:22:39.207073 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:39.207042 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc" Sep 10 12:22:39.495890 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:39.495864 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc"] Sep 10 12:22:39.498184 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:22:39.498136 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9030356a_4c10_4a92_a97d_2d884c6ef400.slice/crio-d1dceb0b66d4a4f831331fcc9172618bd301bf7e56e2cc0d0d099aae36173420 WatchSource:0}: Error finding container d1dceb0b66d4a4f831331fcc9172618bd301bf7e56e2cc0d0d099aae36173420: Status 404 returned error can't find the container with id d1dceb0b66d4a4f831331fcc9172618bd301bf7e56e2cc0d0d099aae36173420 Sep 10 12:22:39.966571 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:39.966537 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc" event={"ID":"9030356a-4c10-4a92-a97d-2d884c6ef400","Type":"ContainerStarted","Data":"4dd0bad2a409608d26c7021df7187e09fffea93565fa3bcde6904c674fc786de"} Sep 10 12:22:39.966971 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:39.966595 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc" event={"ID":"9030356a-4c10-4a92-a97d-2d884c6ef400","Type":"ContainerStarted","Data":"d1dceb0b66d4a4f831331fcc9172618bd301bf7e56e2cc0d0d099aae36173420"} Sep 10 12:22:39.986493 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:39.986447 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc" podStartSLOduration=1.9864324199999999 podStartE2EDuration="1.98643242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:22:39.984211138 +0000 UTC m=+1867.863600570" watchObservedRunningTime="2026-09-10 12:22:39.98643242 +0000 UTC m=+1867.865821851" Sep 10 12:22:40.979977 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:40.979945 2582 generic.go:356] "Generic (PLEG): container finished" podID="9030356a-4c10-4a92-a97d-2d884c6ef400" containerID="4dd0bad2a409608d26c7021df7187e09fffea93565fa3bcde6904c674fc786de" exitCode=0 Sep 10 12:22:40.980399 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:40.980035 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc" event={"ID":"9030356a-4c10-4a92-a97d-2d884c6ef400","Type":"ContainerDied","Data":"4dd0bad2a409608d26c7021df7187e09fffea93565fa3bcde6904c674fc786de"} Sep 10 12:22:42.121828 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:42.121805 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc" Sep 10 12:22:42.223634 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:42.223603 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9030356a-4c10-4a92-a97d-2d884c6ef400-maas-api-service-ca\") pod \"9030356a-4c10-4a92-a97d-2d884c6ef400\" (UID: \"9030356a-4c10-4a92-a97d-2d884c6ef400\") " Sep 10 12:22:42.223972 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:42.223948 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9030356a-4c10-4a92-a97d-2d884c6ef400-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9030356a-4c10-4a92-a97d-2d884c6ef400" (UID: "9030356a-4c10-4a92-a97d-2d884c6ef400"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:22:42.324631 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:42.324538 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9030356a-4c10-4a92-a97d-2d884c6ef400-maas-api-service-ca\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:22:42.992475 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:42.992444 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc" Sep 10 12:22:42.992648 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:42.992482 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc" event={"ID":"9030356a-4c10-4a92-a97d-2d884c6ef400","Type":"ContainerDied","Data":"d1dceb0b66d4a4f831331fcc9172618bd301bf7e56e2cc0d0d099aae36173420"} Sep 10 12:22:42.992648 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:42.992527 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1dceb0b66d4a4f831331fcc9172618bd301bf7e56e2cc0d0d099aae36173420" Sep 10 12:22:49.593539 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:49.593504 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94"] Sep 10 12:22:49.597710 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:49.597675 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ed5ac-29817375-lpr94"] Sep 10 12:22:49.941049 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:49.941019 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc"] Sep 10 12:22:49.946061 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:49.946035 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7ed5ac-rh4fc"] Sep 10 12:22:50.630222 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:50.630188 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9030356a-4c10-4a92-a97d-2d884c6ef400" path="/var/lib/kubelet/pods/9030356a-4c10-4a92-a97d-2d884c6ef400/volumes" Sep 10 12:22:50.630596 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:50.630489 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaf4c770-1f43-4eb6-8699-13e0463b30bd" path="/var/lib/kubelet/pods/eaf4c770-1f43-4eb6-8699-13e0463b30bd/volumes" Sep 10 12:22:52.266301 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:52.266271 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:22:57.154766 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:57.154686 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2"] Sep 10 12:22:57.155211 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:22:57.154895 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerName="istio-proxy" containerID="cri-o://b149175f6b64d0d1abd1767a2f6942ac826672ef88fee96c6af3255191e9a8ce" gracePeriod=30 Sep 10 12:23:01.530442 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:01.530409 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:23:02.410876 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.410850 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:23:02.515089 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.515048 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-credential-socket\") pod \"eba69eaf-fc89-4c26-889a-c45223e69cc7\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " Sep 10 12:23:02.515302 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.515175 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-token\") pod \"eba69eaf-fc89-4c26-889a-c45223e69cc7\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " Sep 10 12:23:02.515302 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.515210 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-workload-certs\") pod \"eba69eaf-fc89-4c26-889a-c45223e69cc7\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " Sep 10 12:23:02.515302 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.515242 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba69eaf-fc89-4c26-889a-c45223e69cc7-istiod-ca-cert\") pod \"eba69eaf-fc89-4c26-889a-c45223e69cc7\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " Sep 10 12:23:02.515302 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.515272 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-workload-socket\") pod \"eba69eaf-fc89-4c26-889a-c45223e69cc7\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " Sep 10 12:23:02.515563 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.515311 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-envoy\") pod \"eba69eaf-fc89-4c26-889a-c45223e69cc7\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " Sep 10 12:23:02.515563 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.515350 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnf9b\" (UniqueName: \"kubernetes.io/projected/eba69eaf-fc89-4c26-889a-c45223e69cc7-kube-api-access-fnf9b\") pod \"eba69eaf-fc89-4c26-889a-c45223e69cc7\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " Sep 10 12:23:02.515563 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.515404 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-data\") pod \"eba69eaf-fc89-4c26-889a-c45223e69cc7\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " Sep 10 12:23:02.515563 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.515473 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-podinfo\") pod \"eba69eaf-fc89-4c26-889a-c45223e69cc7\" (UID: \"eba69eaf-fc89-4c26-889a-c45223e69cc7\") " Sep 10 12:23:02.515563 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.515536 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eba69eaf-fc89-4c26-889a-c45223e69cc7" (UID: "eba69eaf-fc89-4c26-889a-c45223e69cc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:23:02.515830 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.515777 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-data" (OuterVolumeSpecName: "istio-data") pod "eba69eaf-fc89-4c26-889a-c45223e69cc7" (UID: "eba69eaf-fc89-4c26-889a-c45223e69cc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:23:02.515830 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.515817 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-workload-certs\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:23:02.516292 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.516266 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eba69eaf-fc89-4c26-889a-c45223e69cc7" (UID: "eba69eaf-fc89-4c26-889a-c45223e69cc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:23:02.516481 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.516431 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba69eaf-fc89-4c26-889a-c45223e69cc7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eba69eaf-fc89-4c26-889a-c45223e69cc7" (UID: "eba69eaf-fc89-4c26-889a-c45223e69cc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:23:02.516703 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.516673 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eba69eaf-fc89-4c26-889a-c45223e69cc7" (UID: "eba69eaf-fc89-4c26-889a-c45223e69cc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:23:02.518370 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.518336 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eba69eaf-fc89-4c26-889a-c45223e69cc7" (UID: "eba69eaf-fc89-4c26-889a-c45223e69cc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:23:02.519037 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.518842 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eba69eaf-fc89-4c26-889a-c45223e69cc7" (UID: "eba69eaf-fc89-4c26-889a-c45223e69cc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 12:23:02.519037 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.518966 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba69eaf-fc89-4c26-889a-c45223e69cc7-kube-api-access-fnf9b" (OuterVolumeSpecName: "kube-api-access-fnf9b") pod "eba69eaf-fc89-4c26-889a-c45223e69cc7" (UID: "eba69eaf-fc89-4c26-889a-c45223e69cc7"). InnerVolumeSpecName "kube-api-access-fnf9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:23:02.519037 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.518982 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-token" (OuterVolumeSpecName: "istio-token") pod "eba69eaf-fc89-4c26-889a-c45223e69cc7" (UID: "eba69eaf-fc89-4c26-889a-c45223e69cc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:23:02.617243 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.617204 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-token\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:23:02.617243 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.617238 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba69eaf-fc89-4c26-889a-c45223e69cc7-istiod-ca-cert\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:23:02.617243 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.617248 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-workload-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:23:02.617865 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.617256 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-envoy\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:23:02.617865 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.617267 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnf9b\" (UniqueName: \"kubernetes.io/projected/eba69eaf-fc89-4c26-889a-c45223e69cc7-kube-api-access-fnf9b\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:23:02.617865 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.617275 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-data\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:23:02.617865 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.617283 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba69eaf-fc89-4c26-889a-c45223e69cc7-istio-podinfo\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:23:02.617865 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:02.617294 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba69eaf-fc89-4c26-889a-c45223e69cc7-credential-socket\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:23:03.103694 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:03.103663 2582 generic.go:356] "Generic (PLEG): container finished" podID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerID="b149175f6b64d0d1abd1767a2f6942ac826672ef88fee96c6af3255191e9a8ce" exitCode=0 Sep 10 12:23:03.103908 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:03.103725 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" Sep 10 12:23:03.103908 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:03.103753 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" event={"ID":"eba69eaf-fc89-4c26-889a-c45223e69cc7","Type":"ContainerDied","Data":"b149175f6b64d0d1abd1767a2f6942ac826672ef88fee96c6af3255191e9a8ce"} Sep 10 12:23:03.103908 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:03.103806 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2" event={"ID":"eba69eaf-fc89-4c26-889a-c45223e69cc7","Type":"ContainerDied","Data":"af171c9c3f9b4a65394fc3940e760ab0e80d425d1432690c7ef0beb9166fac3a"} Sep 10 12:23:03.103908 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:03.103828 2582 scope.go:117] "RemoveContainer" containerID="b149175f6b64d0d1abd1767a2f6942ac826672ef88fee96c6af3255191e9a8ce" Sep 10 12:23:03.113193 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:03.113169 2582 scope.go:117] "RemoveContainer" containerID="38b9e8f2b1e0589dc1917e31676ad2b7abc8ca76777fd8dde17d104ca8cb3af7" Sep 10 12:23:03.122234 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:03.122210 2582 scope.go:117] "RemoveContainer" containerID="b149175f6b64d0d1abd1767a2f6942ac826672ef88fee96c6af3255191e9a8ce" Sep 10 12:23:03.123226 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:23:03.123193 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b149175f6b64d0d1abd1767a2f6942ac826672ef88fee96c6af3255191e9a8ce\": container with ID starting with b149175f6b64d0d1abd1767a2f6942ac826672ef88fee96c6af3255191e9a8ce not found: ID does not exist" containerID="b149175f6b64d0d1abd1767a2f6942ac826672ef88fee96c6af3255191e9a8ce" Sep 10 12:23:03.123345 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:03.123239 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b149175f6b64d0d1abd1767a2f6942ac826672ef88fee96c6af3255191e9a8ce"} err="failed to get container status \"b149175f6b64d0d1abd1767a2f6942ac826672ef88fee96c6af3255191e9a8ce\": rpc error: code = NotFound desc = could not find container \"b149175f6b64d0d1abd1767a2f6942ac826672ef88fee96c6af3255191e9a8ce\": container with ID starting with b149175f6b64d0d1abd1767a2f6942ac826672ef88fee96c6af3255191e9a8ce not found: ID does not exist" Sep 10 12:23:03.123345 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:03.123261 2582 scope.go:117] "RemoveContainer" containerID="38b9e8f2b1e0589dc1917e31676ad2b7abc8ca76777fd8dde17d104ca8cb3af7" Sep 10 12:23:03.123583 ip-10-0-136-197 kubenswrapper[2582]: E0910 12:23:03.123557 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b9e8f2b1e0589dc1917e31676ad2b7abc8ca76777fd8dde17d104ca8cb3af7\": container with ID starting with 38b9e8f2b1e0589dc1917e31676ad2b7abc8ca76777fd8dde17d104ca8cb3af7 not found: ID does not exist" containerID="38b9e8f2b1e0589dc1917e31676ad2b7abc8ca76777fd8dde17d104ca8cb3af7" Sep 10 12:23:03.123696 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:03.123590 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b9e8f2b1e0589dc1917e31676ad2b7abc8ca76777fd8dde17d104ca8cb3af7"} err="failed to get container status \"38b9e8f2b1e0589dc1917e31676ad2b7abc8ca76777fd8dde17d104ca8cb3af7\": rpc error: code = NotFound desc = could not find container \"38b9e8f2b1e0589dc1917e31676ad2b7abc8ca76777fd8dde17d104ca8cb3af7\": container with ID starting with 38b9e8f2b1e0589dc1917e31676ad2b7abc8ca76777fd8dde17d104ca8cb3af7 not found: ID does not exist" Sep 10 12:23:03.125008 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:03.124975 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2"] Sep 10 12:23:03.130221 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:03.130199 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7ed5ac-openshift-default-878c8488d-9zmr2"] Sep 10 12:23:04.630804 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:04.630776 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" path="/var/lib/kubelet/pods/eba69eaf-fc89-4c26-889a-c45223e69cc7/volumes" Sep 10 12:23:33.087040 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:33.087007 2582 scope.go:117] "RemoveContainer" containerID="ec5116ba0c39cb0363a2c079f34c031100d383237a74fd6ec92eec618828958d" Sep 10 12:23:33.095915 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:33.095897 2582 scope.go:117] "RemoveContainer" containerID="c6be41bd6a779c6a2e8777f1468d2c150899f1e035f07d450f6b3baea38f2129" Sep 10 12:23:33.103623 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:33.103603 2582 scope.go:117] "RemoveContainer" containerID="a31e2de93e756a84aacf0932fe647620038a453653dd9c8cade929847deaed26" Sep 10 12:23:33.110996 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:23:33.110979 2582 scope.go:117] "RemoveContainer" containerID="ae401c9908bc26f783685f02e7bef2ac87e04688a73e180c3de2b2ce7bebf114" Sep 10 12:24:23.095859 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:24:23.095817 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:24:48.181617 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:24:48.181576 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:24:55.977824 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:24:55.977791 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:25:03.286473 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:03.286434 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:25:12.006480 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:12.006437 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:25:26.102130 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:26.102090 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:25:34.564314 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:34.564278 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:25:41.828028 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:41.827988 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm"] Sep 10 12:25:41.828587 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:41.828217 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" podUID="5b61d2f4-ecfc-465a-a2f0-5849df3e0608" containerName="manager" containerID="cri-o://6b5faac52e41619ad76d73d7a35f88f3c466b1afd4f56e91f2d14b9bb3d1a9e6" gracePeriod=10 Sep 10 12:25:41.945811 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:41.945780 2582 generic.go:356] "Generic (PLEG): container finished" podID="5b61d2f4-ecfc-465a-a2f0-5849df3e0608" containerID="6b5faac52e41619ad76d73d7a35f88f3c466b1afd4f56e91f2d14b9bb3d1a9e6" exitCode=0 Sep 10 12:25:41.945966 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:41.945863 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" event={"ID":"5b61d2f4-ecfc-465a-a2f0-5849df3e0608","Type":"ContainerDied","Data":"6b5faac52e41619ad76d73d7a35f88f3c466b1afd4f56e91f2d14b9bb3d1a9e6"} Sep 10 12:25:42.089328 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:42.089268 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" Sep 10 12:25:42.142075 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:42.142043 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kdhk\" (UniqueName: \"kubernetes.io/projected/5b61d2f4-ecfc-465a-a2f0-5849df3e0608-kube-api-access-7kdhk\") pod \"5b61d2f4-ecfc-465a-a2f0-5849df3e0608\" (UID: \"5b61d2f4-ecfc-465a-a2f0-5849df3e0608\") " Sep 10 12:25:42.142251 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:42.142123 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b61d2f4-ecfc-465a-a2f0-5849df3e0608-extensions-socket-volume\") pod \"5b61d2f4-ecfc-465a-a2f0-5849df3e0608\" (UID: \"5b61d2f4-ecfc-465a-a2f0-5849df3e0608\") " Sep 10 12:25:42.142515 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:42.142491 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b61d2f4-ecfc-465a-a2f0-5849df3e0608-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5b61d2f4-ecfc-465a-a2f0-5849df3e0608" (UID: "5b61d2f4-ecfc-465a-a2f0-5849df3e0608"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:25:42.144516 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:42.144490 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b61d2f4-ecfc-465a-a2f0-5849df3e0608-kube-api-access-7kdhk" (OuterVolumeSpecName: "kube-api-access-7kdhk") pod "5b61d2f4-ecfc-465a-a2f0-5849df3e0608" (UID: "5b61d2f4-ecfc-465a-a2f0-5849df3e0608"). InnerVolumeSpecName "kube-api-access-7kdhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:25:42.243209 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:42.243170 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kdhk\" (UniqueName: \"kubernetes.io/projected/5b61d2f4-ecfc-465a-a2f0-5849df3e0608-kube-api-access-7kdhk\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:25:42.243209 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:42.243205 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b61d2f4-ecfc-465a-a2f0-5849df3e0608-extensions-socket-volume\") on node \"ip-10-0-136-197.ec2.internal\" DevicePath \"\"" Sep 10 12:25:42.953164 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:42.953119 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" Sep 10 12:25:42.953164 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:42.953116 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm" event={"ID":"5b61d2f4-ecfc-465a-a2f0-5849df3e0608","Type":"ContainerDied","Data":"9a787fa9fd278bd0f135edb1a34fa0f788c111b100b905ca44cb178282b82183"} Sep 10 12:25:42.953614 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:42.953225 2582 scope.go:117] "RemoveContainer" containerID="6b5faac52e41619ad76d73d7a35f88f3c466b1afd4f56e91f2d14b9bb3d1a9e6" Sep 10 12:25:42.980437 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:42.980414 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm"] Sep 10 12:25:42.986035 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:42.986009 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-t89fm"] Sep 10 12:25:44.629798 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:25:44.629764 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b61d2f4-ecfc-465a-a2f0-5849df3e0608" path="/var/lib/kubelet/pods/5b61d2f4-ecfc-465a-a2f0-5849df3e0608/volumes" Sep 10 12:26:32.777464 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:32.777437 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 12:26:32.779349 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:32.779326 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gnz8p_b98c0ee8-844a-4f6c-89ce-f9492e87509b/ovn-acl-logging/0.log" Sep 10 12:26:42.542588 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.542550 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55"] Sep 10 12:26:42.543072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.542948 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9030356a-4c10-4a92-a97d-2d884c6ef400" containerName="revoke-keys" Sep 10 12:26:42.543072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.542959 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9030356a-4c10-4a92-a97d-2d884c6ef400" containerName="revoke-keys" Sep 10 12:26:42.543072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.542972 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b61d2f4-ecfc-465a-a2f0-5849df3e0608" containerName="manager" Sep 10 12:26:42.543072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.542977 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b61d2f4-ecfc-465a-a2f0-5849df3e0608" containerName="manager" Sep 10 12:26:42.543072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.542986 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" Sep 10 12:26:42.543072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.542992 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe6f4c64-aec6-4bc4-bbee-4534b213345e" containerName="istio-proxy" Sep 10 12:26:42.543072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.542999 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerName="istio-proxy" Sep 10 12:26:42.543072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.543004 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerName="istio-proxy" Sep 10 12:26:42.543072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.543009 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerName="istio-proxy" Sep 10 12:26:42.543072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.543015 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerName="istio-proxy" Sep 10 12:26:42.543072 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.543078 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b61d2f4-ecfc-465a-a2f0-5849df3e0608" containerName="manager" Sep 10 12:26:42.543472 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.543087 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerName="istio-proxy" Sep 10 12:26:42.543472 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.543092 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba69eaf-fc89-4c26-889a-c45223e69cc7" containerName="istio-proxy" Sep 10 12:26:42.543472 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.543098 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9030356a-4c10-4a92-a97d-2d884c6ef400" containerName="revoke-keys" Sep 10 12:26:42.546262 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.546246 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55" Sep 10 12:26:42.550070 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.550045 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4mtmh\"" Sep 10 12:26:42.570596 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.570568 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55"] Sep 10 12:26:42.681265 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.681238 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2zxc\" (UniqueName: \"kubernetes.io/projected/d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8-kube-api-access-v2zxc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-crv55\" (UID: \"d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55" Sep 10 12:26:42.681444 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.681300 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-crv55\" (UID: \"d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55" Sep 10 12:26:42.781918 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.781883 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2zxc\" (UniqueName: \"kubernetes.io/projected/d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8-kube-api-access-v2zxc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-crv55\" (UID: \"d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55" Sep 10 12:26:42.782117 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.781959 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-crv55\" (UID: \"d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55" Sep 10 12:26:42.782363 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.782342 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-crv55\" (UID: \"d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55" Sep 10 12:26:42.796992 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.796920 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2zxc\" (UniqueName: \"kubernetes.io/projected/d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8-kube-api-access-v2zxc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-crv55\" (UID: \"d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55" Sep 10 12:26:42.856800 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:42.856774 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55" Sep 10 12:26:43.170849 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:43.170695 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55"] Sep 10 12:26:43.173985 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:26:43.173958 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d9f2d9_089e_42d3_9c7f_3ba91a1d0bb8.slice/crio-473adc1fda4193ddb6d544ac018c61bd3fc7b7953c813e22db3c755bb6331cf8 WatchSource:0}: Error finding container 473adc1fda4193ddb6d544ac018c61bd3fc7b7953c813e22db3c755bb6331cf8: Status 404 returned error can't find the container with id 473adc1fda4193ddb6d544ac018c61bd3fc7b7953c813e22db3c755bb6331cf8 Sep 10 12:26:43.280443 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:43.280408 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55" event={"ID":"d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8","Type":"ContainerStarted","Data":"51e765a6f50477d6a65a4db5cca5b1a27ccc428b65669efa79778156c801bbb8"} Sep 10 12:26:43.280601 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:43.280465 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55" event={"ID":"d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8","Type":"ContainerStarted","Data":"473adc1fda4193ddb6d544ac018c61bd3fc7b7953c813e22db3c755bb6331cf8"} Sep 10 12:26:43.280601 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:43.280551 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55" Sep 10 12:26:43.330974 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:43.330920 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55" podStartSLOduration=1.330902284 podStartE2EDuration="1.330902284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:26:43.330565758 +0000 UTC m=+2111.209955189" watchObservedRunningTime="2026-09-10 12:26:43.330902284 +0000 UTC m=+2111.210291712" Sep 10 12:26:54.289561 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:26:54.289531 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-crv55" Sep 10 12:27:03.662885 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:27:03.662851 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:27:10.265034 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:27:10.264992 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:27:20.189777 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:27:20.189746 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:27:33.174963 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:27:33.174887 2582 scope.go:117] "RemoveContainer" containerID="a7c518c9dc36ac08c703ae7fbd10b9296ce03b39ecf9d8718a35e98753de0daf" Sep 10 12:27:33.183573 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:27:33.183554 2582 scope.go:117] "RemoveContainer" containerID="fe6e4fd355b8cd6b71be1878f3bf199ed42d10b8a351e187996137b1428def73" Sep 10 12:27:34.682395 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:27:34.682361 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:27:42.011893 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:27:42.011858 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:28:03.057788 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:03.057747 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:28:12.258976 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:12.258938 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:28:20.066659 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:20.066619 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:28:27.981456 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:27.981420 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:28:33.207498 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:33.207466 2582 scope.go:117] "RemoveContainer" containerID="b010a7a22ee8444a240466ae0c917c75009bee0e4639da1adedfd3902966918f" Sep 10 12:28:33.216138 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:33.216121 2582 scope.go:117] "RemoveContainer" containerID="e789ac4bdeb3a48c1fc4458a7486dac959de5707e35f7d602c82856857467bd9" Sep 10 12:28:34.764249 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:34.764209 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:28:40.662826 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:40.662784 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:28:47.595600 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:47.595566 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-54pt4"] Sep 10 12:28:50.398676 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:50.398637 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-687c5c4ddc-lgd7p_e2244333-af65-4c1e-8ebb-8df9410490c1/authorino/0.log" Sep 10 12:28:55.824922 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:55.824841 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-66rzq_446e89f1-88a5-4676-a504-fd413ded2aa1/manager/0.log" Sep 10 12:28:55.942409 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:55.942380 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-px45j_01ccad84-896c-4704-85ab-3dee8a5caf02/manager/0.log" Sep 10 12:28:56.188389 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:56.188368 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-cfkvr_37b9575a-5b55-4dfd-b70b-a254bedee0c8/server/0.log" Sep 10 12:28:56.307332 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:56.307301 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9d78j_3650596d-f3bb-4d38-8c6c-07c6fd3a5407/manager/0.log" Sep 10 12:28:56.570097 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:56.570021 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d86dfbd7c-58rlx_649bbbfd-c48f-4a44-b7bc-e0fbb0815fee/manager/0.log" Sep 10 12:28:56.577789 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:56.577771 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d86dfbd7c-58rlx_649bbbfd-c48f-4a44-b7bc-e0fbb0815fee/copy-manifests/0.log" Sep 10 12:28:57.263666 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:57.263638 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817375-bv82x_dd640259-5600-4652-8b01-c0c6e9dab37c/cleanup/0.log" Sep 10 12:28:58.203438 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:58.203410 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-687c5c4ddc-lgd7p_e2244333-af65-4c1e-8ebb-8df9410490c1/authorino/0.log" Sep 10 12:28:58.332190 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:58.332160 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-pkw6v_7da9e8ee-1931-4828-9aba-4f17f98cbd65/manager/0.log" Sep 10 12:28:58.452369 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:58.452335 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-5sn2n_a99e6748-3296-4f88-bf0d-44b7370d72bb/manager/0.log" Sep 10 12:28:58.701516 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:58.701492 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-crv55_d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8/manager/0.log" Sep 10 12:28:58.822397 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:58.822369 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-54pt4_35563073-52de-46b6-913e-da70e41d9355/limitador/0.log" Sep 10 12:28:58.949267 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:58.949241 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-z65zz_1f556bb6-6163-4685-afcf-d1151a4dbb9d/manager/0.log" Sep 10 12:28:59.556378 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:59.556348 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5986dd8d8b-tx97j_ae94e46e-b2f2-4d01-971c-81b2866a4686/kube-auth-proxy/0.log" Sep 10 12:28:59.798438 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:28:59.798406 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-thwsf_13ee911d-d862-4efa-b0c0-7bf76dbabaab/istio-proxy/1.log" Sep 10 12:29:00.741052 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:00.741023 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-cklvd_898633e3-8f5b-4a41-98fc-81bdbd50bd21/main/0.log" Sep 10 12:29:00.858560 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:00.858531 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-d57w5_f541f120-87ff-4a37-a56f-365e603cf2d9/main/0.log" Sep 10 12:29:01.210020 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:01.209992 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-68kk8_35684a10-f5ad-4ef9-9c2c-af9a300a02d6/main/0.log" Sep 10 12:29:06.736076 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:06.736046 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8xpjf/must-gather-949br"] Sep 10 12:29:06.739630 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:06.739598 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xpjf/must-gather-949br" Sep 10 12:29:06.742365 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:06.742347 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8xpjf\"/\"default-dockercfg-k9vmw\"" Sep 10 12:29:06.743519 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:06.743500 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xpjf\"/\"kube-root-ca.crt\"" Sep 10 12:29:06.743576 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:06.743513 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xpjf\"/\"openshift-service-ca.crt\"" Sep 10 12:29:06.756096 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:06.756076 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xpjf/must-gather-949br"] Sep 10 12:29:06.866558 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:06.866528 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2fx\" (UniqueName: \"kubernetes.io/projected/cf98d216-0e62-415b-8aa4-e26aae0ca516-kube-api-access-pq2fx\") pod \"must-gather-949br\" (UID: \"cf98d216-0e62-415b-8aa4-e26aae0ca516\") " pod="openshift-must-gather-8xpjf/must-gather-949br" Sep 10 12:29:06.866709 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:06.866621 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf98d216-0e62-415b-8aa4-e26aae0ca516-must-gather-output\") pod \"must-gather-949br\" (UID: \"cf98d216-0e62-415b-8aa4-e26aae0ca516\") " pod="openshift-must-gather-8xpjf/must-gather-949br" Sep 10 12:29:06.974481 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:06.974428 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf98d216-0e62-415b-8aa4-e26aae0ca516-must-gather-output\") pod \"must-gather-949br\" (UID: \"cf98d216-0e62-415b-8aa4-e26aae0ca516\") " pod="openshift-must-gather-8xpjf/must-gather-949br" Sep 10 12:29:06.974654 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:06.974528 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2fx\" (UniqueName: \"kubernetes.io/projected/cf98d216-0e62-415b-8aa4-e26aae0ca516-kube-api-access-pq2fx\") pod \"must-gather-949br\" (UID: \"cf98d216-0e62-415b-8aa4-e26aae0ca516\") " pod="openshift-must-gather-8xpjf/must-gather-949br" Sep 10 12:29:06.974945 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:06.974920 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf98d216-0e62-415b-8aa4-e26aae0ca516-must-gather-output\") pod \"must-gather-949br\" (UID: \"cf98d216-0e62-415b-8aa4-e26aae0ca516\") " pod="openshift-must-gather-8xpjf/must-gather-949br" Sep 10 12:29:06.983911 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:06.983890 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2fx\" (UniqueName: \"kubernetes.io/projected/cf98d216-0e62-415b-8aa4-e26aae0ca516-kube-api-access-pq2fx\") pod \"must-gather-949br\" (UID: \"cf98d216-0e62-415b-8aa4-e26aae0ca516\") " pod="openshift-must-gather-8xpjf/must-gather-949br" Sep 10 12:29:07.048980 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:07.048908 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xpjf/must-gather-949br" Sep 10 12:29:07.319735 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:07.319710 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xpjf/must-gather-949br"] Sep 10 12:29:07.321298 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:29:07.321272 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf98d216_0e62_415b_8aa4_e26aae0ca516.slice/crio-3f5f9552e22aecf92ca4cadf1e3b64d8d81f5eaae613cd0c4e2d0c9f0f746efd WatchSource:0}: Error finding container 3f5f9552e22aecf92ca4cadf1e3b64d8d81f5eaae613cd0c4e2d0c9f0f746efd: Status 404 returned error can't find the container with id 3f5f9552e22aecf92ca4cadf1e3b64d8d81f5eaae613cd0c4e2d0c9f0f746efd Sep 10 12:29:07.323113 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:07.323095 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 12:29:08.047382 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:08.047333 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xpjf/must-gather-949br" event={"ID":"cf98d216-0e62-415b-8aa4-e26aae0ca516","Type":"ContainerStarted","Data":"3f5f9552e22aecf92ca4cadf1e3b64d8d81f5eaae613cd0c4e2d0c9f0f746efd"} Sep 10 12:29:09.056242 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:09.056198 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xpjf/must-gather-949br" event={"ID":"cf98d216-0e62-415b-8aa4-e26aae0ca516","Type":"ContainerStarted","Data":"0200a28db78f8a81cd21a4c5ed0082fb3ba8ae2b0fa50275a639451a1ad42e50"} Sep 10 12:29:09.056624 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:09.056257 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xpjf/must-gather-949br" event={"ID":"cf98d216-0e62-415b-8aa4-e26aae0ca516","Type":"ContainerStarted","Data":"7b20de740da5314f3e7c11407e2d5e692e6e2707001e6a29d8c9ccab9e301a9c"} Sep 10 12:29:09.085718 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:09.085669 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8xpjf/must-gather-949br" podStartSLOduration=2.288139142 podStartE2EDuration="3.085640683s" totalImagesPullingTime="797.501541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:29:09.084509894 +0000 UTC m=+2256.963899328" watchObservedRunningTime="2026-09-10 12:29:09.085640683 +0000 UTC m=+2256.965030119" Sep 10 12:29:09.822894 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:09.822861 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lkhnc_62e51f21-9e30-4c69-bca0-b1aca76a0a24/global-pull-secret-syncer/0.log" Sep 10 12:29:09.993988 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:09.993954 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wjzd5_483e138d-49da-44ef-9e55-0d50f70ffb75/konnectivity-agent/0.log" Sep 10 12:29:10.078775 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:10.078688 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-197.ec2.internal_03d9f1651513641d55fc001cd4afe684/haproxy/0.log" Sep 10 12:29:14.796032 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:14.795999 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-687c5c4ddc-lgd7p_e2244333-af65-4c1e-8ebb-8df9410490c1/authorino/0.log" Sep 10 12:29:14.890022 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:14.889985 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-pkw6v_7da9e8ee-1931-4828-9aba-4f17f98cbd65/manager/0.log" Sep 10 12:29:14.966176 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:14.966126 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-5sn2n_a99e6748-3296-4f88-bf0d-44b7370d72bb/manager/0.log" Sep 10 12:29:15.085995 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:15.085913 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-crv55_d3d9f2d9-089e-42d3-9c7f-3ba91a1d0bb8/manager/0.log" Sep 10 12:29:15.139488 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:15.139449 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-54pt4_35563073-52de-46b6-913e-da70e41d9355/limitador/0.log" Sep 10 12:29:15.249271 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:15.247812 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-z65zz_1f556bb6-6163-4685-afcf-d1151a4dbb9d/manager/0.log" Sep 10 12:29:17.013953 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:17.013914 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-bc4dh_6b705d50-4f27-46cb-943e-94c50105943e/monitoring-plugin/0.log" Sep 10 12:29:17.101247 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:17.101165 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-crr6h_77c638dd-4da7-41bf-8e1f-d6d3d773c54c/node-exporter/0.log" Sep 10 12:29:17.151862 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:17.151833 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-crr6h_77c638dd-4da7-41bf-8e1f-d6d3d773c54c/kube-rbac-proxy/0.log" Sep 10 12:29:17.205829 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:17.205793 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-crr6h_77c638dd-4da7-41bf-8e1f-d6d3d773c54c/init-textfile/0.log" Sep 10 12:29:17.993377 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:17.993347 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-74qg6_5899cf32-383d-48b7-a082-c7b36ae1e145/prometheus-operator/0.log" Sep 10 12:29:18.025134 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:18.025094 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-74qg6_5899cf32-383d-48b7-a082-c7b36ae1e145/kube-rbac-proxy/0.log" Sep 10 12:29:18.060267 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:18.060233 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-67dpz_a847cea1-5939-4dbf-b6db-4abdc5a99b6e/prometheus-operator-admission-webhook/0.log" Sep 10 12:29:18.101422 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:18.101391 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-776865cb97-cc5bx_bf999d19-c4f2-472c-b8ec-bb1ac70195d4/telemeter-client/0.log" Sep 10 12:29:18.134424 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:18.134393 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-776865cb97-cc5bx_bf999d19-c4f2-472c-b8ec-bb1ac70195d4/reload/0.log" Sep 10 12:29:18.165522 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:18.165493 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-776865cb97-cc5bx_bf999d19-c4f2-472c-b8ec-bb1ac70195d4/kube-rbac-proxy/0.log" Sep 10 12:29:19.253451 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.253416 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r"] Sep 10 12:29:19.259773 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.259743 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.273475 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.273449 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r"] Sep 10 12:29:19.423131 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.423090 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd501f0e-9565-441b-9d4e-2202e48667c4-lib-modules\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.423437 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.423415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd501f0e-9565-441b-9d4e-2202e48667c4-podres\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.423630 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.423617 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd501f0e-9565-441b-9d4e-2202e48667c4-sys\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.423726 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.423713 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fv2z\" (UniqueName: \"kubernetes.io/projected/dd501f0e-9565-441b-9d4e-2202e48667c4-kube-api-access-9fv2z\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.423829 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.423817 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd501f0e-9565-441b-9d4e-2202e48667c4-proc\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.525582 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.525502 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd501f0e-9565-441b-9d4e-2202e48667c4-lib-modules\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.525748 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.525582 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd501f0e-9565-441b-9d4e-2202e48667c4-podres\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.525748 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.525696 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd501f0e-9565-441b-9d4e-2202e48667c4-lib-modules\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.525748 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.525720 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd501f0e-9565-441b-9d4e-2202e48667c4-podres\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.525748 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.525727 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd501f0e-9565-441b-9d4e-2202e48667c4-sys\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.525896 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.525754 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fv2z\" (UniqueName: \"kubernetes.io/projected/dd501f0e-9565-441b-9d4e-2202e48667c4-kube-api-access-9fv2z\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.525896 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.525799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd501f0e-9565-441b-9d4e-2202e48667c4-proc\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.525896 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.525814 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd501f0e-9565-441b-9d4e-2202e48667c4-sys\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.525994 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.525911 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd501f0e-9565-441b-9d4e-2202e48667c4-proc\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.535691 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.535650 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fv2z\" (UniqueName: \"kubernetes.io/projected/dd501f0e-9565-441b-9d4e-2202e48667c4-kube-api-access-9fv2z\") pod \"perf-node-gather-daemonset-gtr5r\" (UID: \"dd501f0e-9565-441b-9d4e-2202e48667c4\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.547392 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.547319 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-xhwjx_1be59f2c-4e40-447c-884e-f32ee3c9af46/networking-console-plugin/0.log" Sep 10 12:29:19.571821 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.571790 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:19.891306 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:19.891275 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r"] Sep 10 12:29:19.893814 ip-10-0-136-197 kubenswrapper[2582]: W0910 12:29:19.893773 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddd501f0e_9565_441b_9d4e_2202e48667c4.slice/crio-59159f464b237320d330c8eafe0ecb8b533ae36cb22bbe714f01df4eedd8b74c WatchSource:0}: Error finding container 59159f464b237320d330c8eafe0ecb8b533ae36cb22bbe714f01df4eedd8b74c: Status 404 returned error can't find the container with id 59159f464b237320d330c8eafe0ecb8b533ae36cb22bbe714f01df4eedd8b74c Sep 10 12:29:20.144851 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:20.144754 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" event={"ID":"dd501f0e-9565-441b-9d4e-2202e48667c4","Type":"ContainerStarted","Data":"0b1debd1d6b051776ae0f2fd2f003f6a7cfa0cd9045bb3ac233718f3d437f80b"} Sep 10 12:29:20.145164 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:20.145120 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" event={"ID":"dd501f0e-9565-441b-9d4e-2202e48667c4","Type":"ContainerStarted","Data":"59159f464b237320d330c8eafe0ecb8b533ae36cb22bbe714f01df4eedd8b74c"} Sep 10 12:29:20.145367 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:20.145349 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:20.168739 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:20.168663 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" podStartSLOduration=1.168640707 podStartE2EDuration="1.168640707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:29:20.168156481 +0000 UTC m=+2268.047545905" watchObservedRunningTime="2026-09-10 12:29:20.168640707 +0000 UTC m=+2268.048030136" Sep 10 12:29:20.936767 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:20.936737 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78df4fdf45-kp5dt_440d2d33-8ff9-422b-8e0c-dbb95296c145/console/0.log" Sep 10 12:29:20.974804 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:20.974776 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-9lg8f_a88118da-c9a3-4593-8a83-5137bdd8984a/download-server/0.log" Sep 10 12:29:21.622277 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:21.622241 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-lwrbw_ccf57465-583d-4ca7-98f4-fb61688670cf/volume-data-source-validator/0.log" Sep 10 12:29:22.685614 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:22.685586 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8927q_fc1da2b0-9c77-4646-8dc1-5aad948d5380/dns/0.log" Sep 10 12:29:22.710868 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:22.710841 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8927q_fc1da2b0-9c77-4646-8dc1-5aad948d5380/kube-rbac-proxy/0.log" Sep 10 12:29:22.864636 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:22.864610 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ddt89_89fde479-2004-4ff5-a4ed-1ad6180e0dda/dns-node-resolver/0.log" Sep 10 12:29:23.499026 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:23.498986 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-d684479cb-4mz2c_aa968658-2fe1-4464-a872-a8eab3c88061/registry/0.log" Sep 10 12:29:23.576006 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:23.575980 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k62hz_5480668d-f15c-4051-a9d4-a2ca9fefe4d3/node-ca/0.log" Sep 10 12:29:24.861177 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:24.861115 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5986dd8d8b-tx97j_ae94e46e-b2f2-4d01-971c-81b2866a4686/kube-auth-proxy/0.log" Sep 10 12:29:24.959916 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:24.959889 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-thwsf_13ee911d-d862-4efa-b0c0-7bf76dbabaab/istio-proxy/0.log" Sep 10 12:29:24.970704 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:24.970675 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-thwsf_13ee911d-d862-4efa-b0c0-7bf76dbabaab/istio-proxy/1.log" Sep 10 12:29:26.016756 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:26.016728 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mk9ks_918767c5-ff0f-4d0a-9dbe-548d0a8463eb/serve-healthcheck-canary/0.log" Sep 10 12:29:26.523847 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:26.523811 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-kwhz7_313f87bb-5d68-4539-a59b-226c262d5309/insights-operator/0.log" Sep 10 12:29:26.524479 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:26.524454 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-kwhz7_313f87bb-5d68-4539-a59b-226c262d5309/insights-operator/1.log" Sep 10 12:29:26.709694 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:26.709663 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wrvdx_a6812340-a7eb-46fc-95de-912e76a052d7/kube-rbac-proxy/0.log" Sep 10 12:29:26.741573 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:26.741547 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wrvdx_a6812340-a7eb-46fc-95de-912e76a052d7/exporter/0.log" Sep 10 12:29:26.775712 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:26.775637 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wrvdx_a6812340-a7eb-46fc-95de-912e76a052d7/extractor/0.log" Sep 10 12:29:27.170048 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:27.169981 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-gtr5r" Sep 10 12:29:29.311531 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:29.311494 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-66rzq_446e89f1-88a5-4676-a504-fd413ded2aa1/manager/0.log" Sep 10 12:29:29.532279 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:29.532243 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-px45j_01ccad84-896c-4704-85ab-3dee8a5caf02/manager/0.log" Sep 10 12:29:29.623087 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:29.623060 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-cfkvr_37b9575a-5b55-4dfd-b70b-a254bedee0c8/server/0.log" Sep 10 12:29:29.723504 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:29.723475 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9d78j_3650596d-f3bb-4d38-8c6c-07c6fd3a5407/manager/0.log" Sep 10 12:29:29.874778 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:29.874664 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d86dfbd7c-58rlx_649bbbfd-c48f-4a44-b7bc-e0fbb0815fee/manager/0.log" Sep 10 12:29:29.892485 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:29.892461 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d86dfbd7c-58rlx_649bbbfd-c48f-4a44-b7bc-e0fbb0815fee/copy-manifests/0.log" Sep 10 12:29:30.639749 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:30.639695 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5986dd8d8b-tx97j_ae94e46e-b2f2-4d01-971c-81b2866a4686/kube-auth-proxy/0.log" Sep 10 12:29:30.681281 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:30.681252 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-thwsf_13ee911d-d862-4efa-b0c0-7bf76dbabaab/istio-proxy/0.log" Sep 10 12:29:30.692802 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:30.692746 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-thwsf_13ee911d-d862-4efa-b0c0-7bf76dbabaab/istio-proxy/1.log" Sep 10 12:29:31.442013 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:31.441985 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-sxxvq_0c05e6c3-2511-42f7-94c9-9dd4cae04099/openshift-lws-operator/0.log" Sep 10 12:29:32.480699 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:32.480668 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5986dd8d8b-tx97j_ae94e46e-b2f2-4d01-971c-81b2866a4686/kube-auth-proxy/0.log" Sep 10 12:29:32.522838 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:32.522812 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-thwsf_13ee911d-d862-4efa-b0c0-7bf76dbabaab/istio-proxy/0.log" Sep 10 12:29:32.533429 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:32.533401 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-thwsf_13ee911d-d862-4efa-b0c0-7bf76dbabaab/istio-proxy/1.log" Sep 10 12:29:33.243964 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:33.243931 2582 scope.go:117] "RemoveContainer" containerID="4dd0bad2a409608d26c7021df7187e09fffea93565fa3bcde6904c674fc786de" Sep 10 12:29:36.596331 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:36.596296 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-gmvjk_1b5f2151-3c8b-4297-9d61-3f978d112702/migrator/0.log" Sep 10 12:29:36.618582 ip-10-0-136-197 kubenswrapper[2582]: I0910 12:29:36.618551 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-gmvjk_1b5f2151-3c8b-4297-9d61-3f978d112702/graceful-termination/0.log"